AI Crawler

Why does treating AI crawlers as one category go wrong?

Because the four jobs have opposite consequences and one switch cannot express them. Blocking a training crawler keeps your writing out of future models and costs you nothing today. Blocking a search crawler removes you from the assistant’s results, which is the outcome almost nobody intends when they say “block AI”. The tokens are separate precisely so the decision can be separate.

What can no AI crawler read?

Anything that only exists after JavaScript runs. Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found the major AI crawlers requesting JavaScript files and executing none of them: 11.5% of GPTBot’s requests were for scripts, and 23.84% of ClaudeBot’s, with no rendering behind either figure.

That single fact outranks everything else on this page. A page whose content arrives client-side is not partly visible to these crawlers, it is absent, and no amount of markup, structure or writing quality changes that. Googlebot renders, so Google’s own surfaces are the exception, which is exactly why a site can look fine in Search Console and be invisible in ChatGPT.

Who else decides whether an AI crawler reaches you?

Your content delivery network, and increasingly by default rather than by your choice. On 1 July 2025 Cloudflare announced it was changing the default to block AI crawlers unless they pay creators, and published the ratio that motivated it: Anthropic’s crawlers were fetching roughly 30,000 pages for every visitor sent back, and OpenAI’s several hundred.

The practical consequence is that robots.txt has stopped being the whole answer. A site can allow every search crawler in its robots.txt and still turn them away at the edge with a 403 or a challenge, with nothing in the file to indicate it. Policy and evidence are two separate checks, and only the second one involves an actual request.

Which of them can reach you today?

Related terms

• GPTBotGPTBot is OpenAI’s training crawler.
• Training DataTraining data is the body of text a language model learns from before it answers anything.
• Robots.txtRobots.txt is a file at a site’s root telling automated crawlers which paths they may fetch.
• Server-Side Rendering (SSR)Server-side rendering means a page’s HTML arrives from the server already containing its content, rather than being assembled in the browser by JavaScript.