# AI Crawler

> Source: https://rankxai.com/glossary/ai-crawler · Last updated: 2026-08-18

An AI crawler is an automated fetcher run by an AI company to collect web pages. AI crawlers do four distinct jobs: training a model, indexing pages for an assistant’s search, fetching a page a user asked about, and, in Google’s case, acting purely as a robots.txt control token.

## Why does treating AI crawlers as one category go wrong?

Because the four jobs have opposite consequences and one switch cannot express them. Blocking a training crawler keeps your writing out of future models and costs you nothing today. Blocking a search crawler removes you from the assistant’s results, which is the outcome almost nobody intends when they say “block AI”. The tokens are separate precisely so the decision can be separate.

The four AI crawler purposes, with one example of each, from the vendors’ own documentation read 18 August 2026.

- Purpose: Training. Example: [GPTBot](/glossary/gptbot). What blocking it costs you: Nothing today. Your content stays out of future model training.
- Purpose: Search indexing. Example: [OAI-SearchBot](/glossary/oai-searchbot). What blocking it costs you: Your pages stop appearing in that assistant’s answers.
- Purpose: User-triggered fetch. Example: [ChatGPT-User](/glossary/chatgpt-user). What blocking it costs you: Little, because several vendors say robots.txt may not apply to it.
- Purpose: Control token only. Example: [Google-Extended](/glossary/google-extended). What blocking it costs you: Gemini training and grounding. Nothing in Google Search.

## What can no AI crawler read?

Anything that only exists after JavaScript runs. Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found the major AI crawlers requesting JavaScript files and executing none of them: 11.5% of GPTBot’s requests were for scripts, and 23.84% of ClaudeBot’s, with no rendering behind either figure.

That single fact outranks everything else on this page. A page whose content arrives client-side is not partly visible to these crawlers, it is absent, and no amount of markup, structure or writing quality changes that. Googlebot renders, so Google’s own surfaces are the exception, which is exactly why a site can look fine in Search Console and be invisible in ChatGPT.

## Who else decides whether an AI crawler reaches you?

Your content delivery network, and increasingly by default rather than by your choice. On 1 July 2025 Cloudflare announced it was [changing the default to block AI crawlers unless they pay creators](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), and published the ratio that motivated it: Anthropic’s crawlers were fetching roughly 30,000 pages for every visitor sent back, and OpenAI’s several hundred.

The practical consequence is that robots.txt has stopped being the whole answer. A site can allow every search crawler in its robots.txt and still turn them away at the edge with a 403 or a challenge, with nothing in the file to indicate it. Policy and evidence are two separate checks, and only the second one involves an actual request.

## Sources

- [OpenAI bot documentation](https://developers.openai.com/api/docs/bots), checked 2026-08-18
- [Cloudflare: Content Independence Day, 1 July 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), checked 2026-08-18
- [Vercel: the rise of the AI crawler, 17 December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026-08-18