ClaudeBot

How does ClaudeBot differ from Claude-SearchBot?

ClaudeBot trains and Claude-SearchBot indexes, the same split OpenAI draws between GPTBot and OAI-SearchBot. The consequence is the same too: disallowing ClaudeBot keeps your writing out of future Anthropic training and leaves Claude able to find and cite your pages. Disallowing Claude-SearchBot is what takes them out of Claude’s search-grounded answers.

Anthropic also supports Crawl-delay, which is not part of the robots.txt standard and which most crawlers ignore. Its documentation gives the example directly, so a site being hit harder than it wants has a lever here that actually works:

User-agent: ClaudeBot
Crawl-delay: 1

Why can a log not tell you which Anthropic bot called?

Because Anthropic publishes one flat address list covering all three bots, at claude.com/crawling/bots.json, and does not publish full user agent strings at all. So an address match confirms a request came from Anthropic and cannot tell you whether it was the training crawler, the search indexer or a user-triggered fetch. That is a real gap, it is Anthropic’s to close, and no tool can work around it honestly.

It matters more than it sounds. The whole per-purpose approach to AI crawler control rests on being able to see which purpose actually visited, and for one major vendor you cannot. Treat any per-bot Claude breakdown you are shown as inference from the user agent string, which is a header anything can send. It also makes the crawl-delay above the one Anthropic control you can confirm is working, because request volume stays visible even when attribution does not.

Does ClaudeBot read JavaScript-rendered content?

No. Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found ClaudeBot requesting JavaScript files in 23.84% of its requests, the highest rate of any crawler measured, and executing none of them. Fetching a script and running it are different things, and the high fetch rate is exactly why people assume otherwise.

The recommendation that follows is worth stating as a position rather than a caveat: if a sentence has to be found, it belongs in the server-rendered HTML. For this class of crawler, client-side rendering is not a small penalty, it is absence. The same measurement is the reason a JavaScript-driven accordion is a worse choice than a native disclosure element for anything that matters: content inside a collapsed element still counts if it is in the HTML, and content injected on click does not exist.

Related terms

• GPTBotGPTBot is OpenAI’s training crawler.
• PerplexityBotPerplexityBot is the crawler behind Perplexity’s search results.
• CCBot (Common Crawl)CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web for anyone to analyse.
• GroundingGrounding is the step that ties a model’s answer to sources it retrieved, so the reply can cite them rather than assert them.