ChatGPT-User

Why does ChatGPT-User ignore robots.txt?

Because the vendors treat a user-triggered fetch as the person browsing rather than as a crawl, and they say so in their own documentation. OpenAI’s wording for ChatGPT-User is that robots.txt rules “may not apply”. Perplexity’s wording for its equivalent is blunter: Perplexity-User “generally ignores robots.txt rules”. Anthropic is the exception and states its bots honour robots.txt without carving out the user-triggered one.

Whether you find that reasonable is a separate question from whether it is true. The practical consequence is the same either way: robots.txt is a request, and for this class of traffic it is a request several vendors have said in advance they may decline.

What follows for anything you actually need to protect?

Nothing behind a robots.txt rule is protected, and that was true long before assistants existed. A page that must not be read by a machine needs authentication, an address rule or a firewall, not a line in a text file that politely asks. User-triggered fetchers have not changed that rule; they have made the gap between the rule and most people’s mental model expensive enough to notice.

For a marketing site the calculation usually runs the other way. A ChatGPT-User fetch means somebody has asked an assistant about your page, which is closer to a visit than to a crawl, and it is the one AI request that maps to a person with an intent. Blocking it is available and rarely what a marketing site wants. There is a middle option most sites skip, which is to allow the fetch and shape what it finds. A page that already answers the question a visitor is likely to be asking about it is a better outcome than a blocked request, and it is the only lever that works at all on traffic robots.txt cannot stop.

How do you recognise ChatGPT-User in a log?

The published string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot, and the addresses are listed at openai.com/chatgpt-user.json. Check the address, not the string. There is no aggregate to look at beyond that. No vendor reports how often a user-triggered fetch happened, so the only count is the one you keep yourself, and on most sites the volume is small enough to disappear into ordinary bot noise unless you filter for it deliberately. If you do count it, count it separately from the crawlers: a ChatGPT-User request tracks a person asking a question right now, so it moves with campaigns and coverage in a way a crawl schedule does not, and averaging the two together hides the only signal in it.

Related terms

• GPTBotGPTBot is OpenAI’s training crawler.
• OAI-SearchBotOAI-SearchBot is the crawler that builds the index behind ChatGPT search.
• PerplexityBotPerplexityBot is the crawler behind Perplexity’s search results.
• ClaudeBotClaudeBot is Anthropic’s training crawler, documented as collecting web content that can contribute to training its models.