# ClaudeBot

> Source: https://rankxai.com/glossary/claudebot · Last updated: 2026-08-18

ClaudeBot is Anthropic’s training crawler, documented as collecting web content that can contribute to training its models. Anthropic runs two others: Claude-SearchBot, which indexes pages so Claude can cite them, and Claude-User, which fetches a page when somebody asks. All three honour robots.txt, which is not true of every vendor.

## How does ClaudeBot differ from Claude-SearchBot?

ClaudeBot trains and Claude-SearchBot indexes, the same split OpenAI draws between [GPTBot](/glossary/gptbot) and [OAI-SearchBot](/glossary/oai-searchbot). The consequence is the same too: disallowing ClaudeBot keeps your writing out of future Anthropic training and leaves Claude able to find and cite your pages. Disallowing Claude-SearchBot is what takes them out of Claude’s search-grounded answers.

Anthropic also supports `Crawl-delay`, which is not part of the robots.txt standard and which most crawlers ignore. Its documentation gives the example directly, so a site being hit harder than it wants has a lever here that actually works:

robots.txt: slow ClaudeBot down rather than blocking it

```
User-agent: ClaudeBot
Crawl-delay: 1
```

## Why can a log not tell you which Anthropic bot called?

Because Anthropic publishes one flat address list covering all three bots, at `claude.com/crawling/bots.json`, and does not publish full user agent strings at all. So an address match confirms a request came from Anthropic and cannot tell you whether it was the training crawler, the search indexer or a user-triggered fetch. That is a real gap, it is Anthropic’s to close, and no tool can work around it honestly.

It matters more than it sounds. The whole per-purpose approach to AI crawler control rests on being able to see which purpose actually visited, and for one major vendor you cannot. Treat any per-bot Claude breakdown you are shown as inference from the user agent string, which is a header anything can send. It also makes the crawl-delay above the one Anthropic control you can confirm is working, because request volume stays visible even when attribution does not.

## Does ClaudeBot read JavaScript-rendered content?

No. Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found ClaudeBot requesting JavaScript files in 23.84% of its requests, the highest rate of any crawler measured, and executing none of them. Fetching a script and running it are different things, and the high fetch rate is exactly why people assume otherwise.

The recommendation that follows is worth stating as a position rather than a caveat: if a sentence has to be found, it belongs in the server-rendered HTML. For this class of crawler, client-side rendering is not a small penalty, it is absence. The same measurement is the reason a JavaScript-driven accordion is a worse choice than a native disclosure element for anything that matters: content inside a collapsed element still counts if it is in the HTML, and content injected on click does not exist.

## Sources

- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518), checked 2026-08-18
- [Vercel: the rise of the AI crawler, 17 December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026-08-18