# Robots.txt

> Source: https://rankxai.com/glossary/robots-txt · Last updated: 2026-08-18

Robots.txt is a file at a site’s root telling automated crawlers which paths they may fetch. Robots.txt controls crawling and not indexing, it is a request rather than an enforcement, and it now carries separate tokens for AI training, AI search and user-triggered fetches.

## Why is robots.txt no longer one decision?

Because every major AI vendor now runs several bots with different jobs, and blocking them as one category produces an outcome almost nobody intends. [GPTBot](/glossary/gptbot) trains, [OAI-SearchBot](/glossary/oai-searchbot) indexes for ChatGPT search, and [ChatGPT-User](/glossary/chatgpt-user) fetches when a person asks. Disallowing the first costs you nothing today; disallowing the second removes you from ChatGPT entirely.

The single most common error in this file is now a wildcard rule written before those tokens existed, or a content delivery network switch labelled “block AI bots” that treats them as one category on your behalf.

## What has robots.txt never done?

Two things people still expect of it. It does not keep a page out of an index: a disallowed URL can still be indexed from links elsewhere, and the control for that is `noindex`, which the crawler has to be allowed to fetch in order to see. Blocking a page in robots.txt and adding a `noindex` to it is a common, self-cancelling pair.

And it has never been access control. Several vendors state directly that their user-triggered fetchers may not apply robots.txt rules, because a human asked. Anything that must not be read needs authentication or a firewall. The file is also public, which people forget when they use it to hide directories. A `Disallow` line naming an admin path or a staging directory is a published list of the places worth looking.

## How should an AI-era robots.txt actually look?

Explicit per purpose, with the search bots conspicuously absent from the block list:

robots.txt: block training, stay visible in AI search

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Not blocked, deliberately: OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, Googlebot.
```

Then check what your server actually returns, because robots.txt is policy and the edge is evidence. A site can allow every search crawler in this file and still turn them away with a 403 it never configured itself. Two syntax notes that cause real damage. The most specific matching group wins rather than the first one, so a narrow rule for a named bot overrides the wildcard group entirely rather than adding to it.

And a bot named in its own group ignores the wildcard group completely, which is how a site ends up allowing something it thought it had blocked twice. Check the file after any content management system update as well. Several platforms rewrite robots.txt on upgrade, and a rule somebody added carefully a year ago can vanish without anybody being told.

## Sources

- [Google Search Central: robots.txt introduction and guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro), checked 2026-08-18
- [OpenAI bot documentation](https://developers.openai.com/api/docs/bots), checked 2026-08-18