Robots.txt

Why is robots.txt no longer one decision?

Because every major AI vendor now runs several bots with different jobs, and blocking them as one category produces an outcome almost nobody intends. GPTBot trains, OAI-SearchBot indexes for ChatGPT search, and ChatGPT-User fetches when a person asks. Disallowing the first costs you nothing today; disallowing the second removes you from ChatGPT entirely.

The single most common error in this file is now a wildcard rule written before those tokens existed, or a content delivery network switch labelled “block AI bots” that treats them as one category on your behalf.

What has robots.txt never done?

Two things people still expect of it. It does not keep a page out of an index: a disallowed URL can still be indexed from links elsewhere, and the control for that is noindex, which the crawler has to be allowed to fetch in order to see. Blocking a page in robots.txt and adding a noindex to it is a common, self-cancelling pair.

And it has never been access control. Several vendors state directly that their user-triggered fetchers may not apply robots.txt rules, because a human asked. Anything that must not be read needs authentication or a firewall. The file is also public, which people forget when they use it to hide directories. A Disallow line naming an admin path or a staging directory is a published list of the places worth looking.

How should an AI-era robots.txt actually look?

Explicit per purpose, with the search bots conspicuously absent from the block list:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Not blocked, deliberately: OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, Googlebot.

Then check what your server actually returns, because robots.txt is policy and the edge is evidence. A site can allow every search crawler in this file and still turn them away with a 403 it never configured itself. Two syntax notes that cause real damage. The most specific matching group wins rather than the first one, so a narrow rule for a named bot overrides the wildcard group entirely rather than adding to it.

And a bot named in its own group ignores the wildcard group completely, which is how a site ends up allowing something it thought it had blocked twice. Check the file after any content management system update as well. Several platforms rewrite robots.txt on upgrade, and a rule somebody added carefully a year ago can vanish without anybody being told.

Does your file do what you think?

Related terms

• AI CrawlerAn AI crawler is an automated fetcher run by an AI company to collect web pages.
• GPTBotGPTBot is OpenAI’s training crawler.
• XML SitemapAn XML sitemap is a file listing the URLs on a site that you want search engines to know about, optionally with the date each was last modified.
• Crawl BudgetCrawl budget is the amount of crawling an engine will do on a site, set by what the server can take and by how much the engine wants the content.