Skip to content
RankX AI
Start Free TrialFree Trial

AI search glossary

CCBot (Common Crawl)

CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web for anyone to analyse. CCBot trains no model itself. CCBot matters to AI visibility because the widely used open training corpora are built by filtering Common Crawl’s archive rather than by crawling sites directly.

Category
AI crawler
Reviewed

Why does a non-profit archive belong in an AI glossary?

Because of what is downstream of it. Common Crawl describes its own mission as “democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone”, and says nothing about AI. But the open datasets model builders actually use are derived from that repository: FineWeb, one of the most widely used, is assembled from 114 Common Crawl snapshots running from 2013 to 2025.

So a Disallow aimed at GPTBot and ClaudeBot but not at CCBot leaves the widest training pathway open, and it is the pathway you can see least of. Nobody publishes which corpora were built from which snapshot, or which model consumed which corpus. The second-order effect is worth planning for as well. A snapshot is a point in time, so what the archive holds about you is whatever your site said on the days it was crawled, and a page you rewrote last year is still in there in its old form. For a brand whose positioning has moved, the archive is a durable record of the previous positioning.

How do you block CCBot, and should you?

One group, with the token Common Crawl publishes. The user agent it sends is CCBot/2.0 (https://commoncrawl.org/faq/).

robots.txt: opt out of the Common Crawl archive
User-agent: CCBot
Disallow: /

Whether you should is a values question rather than a marketing one, and the honest answer cuts both ways. Common Crawl is the closest thing the open web has to a public archive, and it is used by researchers who are building no product at all. Blocking CCBot removes you from that too, and it is not retroactive in either direction: snapshots already published stay published.

When did the snapshots start, and can one be removed?

Common Crawl has been publishing since 2013, and the archive is cumulative rather than a rolling window: FineWeb, built on top of it, draws on 114 separate snapshots running from CC-MAIN-2013-20 to CC-MAIN-2025-26. A page that was public in 2016 is in the archive whether or not the site still exists.

That is the part worth understanding before adding a Disallow. Blocking CCBot stops future collection and removes nothing already published, and there is no takedown route through the crawler. It also means a robots.txt change made today reaches no model that has already been trained, which is the same forward-only limit that applies to every training crawler on this list.

  • GPTBotGPTBot is OpenAI’s training crawler.
  • ClaudeBotClaudeBot is Anthropic’s training crawler, documented as collecting web content that can contribute to training its models.
  • Google-ExtendedGoogle-Extended is a robots.txt token, not a crawler.
  • LLMO (Large Language Model Optimization)LLMO, or Large Language Model Optimization, is a third name for the practice of getting a brand cited by AI assistants.

This definition of CCBot (Common Crawl) was checked against the following sources. Definitions are reviewed quarterly and edited only when they are wrong, so the reviewed date above moves on a real correction and not on a schedule.

Every entry is listed on the RankX AI glossary index, and this page is available as Markdown at /glossary/ccbot.md.

Start here

See where you show up in AI answers today.

Add your site and RankX AI suggests the prompts to track, monitors the keywords that matter and audits your pages, with your first results minutes after you finish setup.

Start Free Trial

7-day free trial. No credit card required. Cancel anytime.