How AI Crawlers Read Your Site

What is an AI crawler?

An AI crawler is a bot that fetches web pages for an artificial intelligence system rather than for a classic search index. AI companies run them for three distinct jobs: collecting web content at scale to train large language models, building the indexes that feed generative search results, and real-time retrieval, the retrieval augmented generation that lets an assistant answer a query with information beyond its training data. Each bot announces itself with a user agent string, which is what every control below keys on.

The three purposes matter more than the names, because the data collection you might object to (LLM training) and the AI crawler activity you probably want (search indexing and retrieval, the requests that surface websites in AI answers) come from different bots with different rules. Blocking by vibes blocks the wrong one.

Which AI crawlers actually matter?

Every major vendor now runs separate crawler bots for separate purposes, and the split is the single most important fact in this topic, because the control you set for one purpose does not apply to the others.

• OpenAI: GPTBot, from OpenAI's model training pipeline, crawls for training; OAI-SearchBot indexes for ChatGPT Search; and ChatGPT-User fetches when a user asks about your URL.
• Anthropic: ClaudeBot crawls for training, Claude-SearchBot indexes for answers, Claude-User fetches on request.
• Perplexity: PerplexityBot builds the Perplexity AI index (rebuilt in-house in 2025), Perplexity-User fetches for individual sessions.
• Google: Googlebot serves everything including AI Overviews and AI Mode; Google-Extended is a control for Gemini training and grounding, and notably does not opt you out of AI Overviews.
• ByteDance: Bytespider crawls for training and is widely reported to ignore robots.txt, which is one reason CDN-level bot controls exist at all.

How do AI crawlers differ from traditional web crawlers?

Traditional search engine crawlers exist to send you human traffic: Googlebot fetches so that search results can rank your pages and readers can click through. Most AI bot traffic has no such loop. A training crawler scrapes web content once and the value flows to model training; Anthropic's crawl-to-referral ratio has been measured in the tens of thousands of fetches per referred visit. Search crawlers from the same AI platforms sit in between: they index so AI assistants can cite you, which is the closest the new bots come to the old bargain.

The operational differences bite too. Decades of search crawling produced bots engineered to avoid overwhelming servers; AI crawler traffic is younger and blunter, with measured traffic spikes and over half of requests re-fetching unchanged pages. And no AI crawler renders JavaScript, where Googlebot does, which is the difference that decides what the LLMs behind the assistants can actually read.

Do AI crawlers execute JavaScript?

No, and the scale of the evidence makes this the most settled fact in AI search. Vercel and MERJ analysed more than 500 million fetches and found GPTBot, ClaudeBot, PerplexityBot and Meta's crawler download JavaScript files, GPTBot in about 11.5 percent of requests and ClaudeBot in about 23.8 percent, and execute none of them. Independent re-verification through 2025 and 2026 found the same: every OpenAI, Anthropic and Perplexity crawler reads raw HTML only. Googlebot remains the one major crawler with full rendering.

The consequence is a one-line test that matters more than any audit: view the page source, not the DOM, and search for the sentence that must be found. Content visible in your browser's inspector but absent from view-source does not exist for most AI systems. Retrieval pipelines then convert that HTML to plain text and hand the model snippets, which is also why nothing meaningful should live only in script tags: a measured test found no engine extracted a price that existed only in JSON-LD.

What about the agentic browsers?

One nuance keeps this from being absolute. AI-powered agent modes, ChatGPT agent, Perplexity Comet, Claude operating a browser, drive real browsers and do render JavaScript. But each of those is a per-user session, an AI agent acting on a page it was already sent to. Retrieval and citation, the systems that decide whether anyone is sent to your page at all, still run on the raw HTML. Build for the crawler; the agent inherits it.

Should you allow or block AI crawlers?

Because the bots split by purpose, the useful question is never should I block AI but which purpose do I want to refuse. The trade per purpose: allowing LLM training bots donates website content to model training with no measured visibility return; allowing the search and retrieval bots is what makes your brand quotable in AI answers, and blocking them is invisibility by choice. Many sites happily block training crawlers while staying retrievable; the reverse mistake, blocking a search-purpose bot to make a point about training, is a self-inflicted removal from the channel this whole discipline is about.

Your CDN may also be deciding how you manage AI access for you. Cloudflare began blocking mixed-use AI crawlers by default for new zones and its free tier from September 2026, and roughly a quarter of the top thousand sites already block GPTBot. If AI visibility matters to you, the check that settles it is not your robots.txt file but your server logs: are the search-purpose bots getting 200 responses in practice.

How to verify what the crawlers can see

1. Run the free AI Crawler Access Checker: it reads your robots.txt the way each named bot does and reports who is allowed, who is blocked, and whether the rules say what you think they say.
2. View source on your key pages and search for the sentences that must be quotable. If they are not in the raw HTML, nothing downstream matters.
3. Check server or CDN logs for the search-purpose bots specifically, OAI-SearchBot, Claude-SearchBot, PerplexityBot, returning 200s. A CDN rule can silently override everything the file says, and the vendors' published IP ranges let you separate verified AI crawler traffic from impostors wearing the same user agent.
4. Expect wasteful bot traffic and cache accordingly: over half of AI crawler requests re-fetch unchanged pages, so serve them cheap cached 200s rather than blocking them for cost.

Everything a crawler needs is also everything extraction needs, so this work compounds: the GEO guide covers what to do with the access once it is verified, and Website Audit runs these checks across the whole site rather than one page at a time.

Sources

• Vercel and MERJ, The Rise of the AI Crawler, 500M+ fetches, checked 20 Aug 2026
• searchVIU, AI crawlers and JavaScript rendering re-verification, checked 20 Aug 2026
• OpenAI bot documentation, checked 20 Aug 2026
• Anthropic crawler documentation, checked 20 Aug 2026
• Perplexity bot documentation, checked 20 Aug 2026
• Cloudflare AI crawler policy, via TechCrunch, July 2026, checked 20 Aug 2026
• searchVIU, what engines really see (JSON-LD-only content unread), checked 20 Aug 2026