# CCBot (Common Crawl)

> Source: https://rankxai.com/glossary/ccbot · Last updated: 2026-08-18

CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web for anyone to analyse. CCBot trains no model itself. CCBot matters to AI visibility because the widely used open training corpora are built by filtering Common Crawl’s archive rather than by crawling sites directly.

## Why does a non-profit archive belong in an AI glossary?

Because of what is downstream of it. Common Crawl describes its own mission as “democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone”, and says nothing about AI. But the open datasets model builders actually use are derived from that repository: FineWeb, one of the most widely used, is assembled from 114 Common Crawl snapshots running from 2013 to 2025.

So a `Disallow` aimed at [GPTBot](/glossary/gptbot) and [ClaudeBot](/glossary/claudebot) but not at CCBot leaves the widest training pathway open, and it is the pathway you can see least of. Nobody publishes which corpora were built from which snapshot, or which model consumed which corpus. The second-order effect is worth planning for as well. A snapshot is a point in time, so what the archive holds about you is whatever your site said on the days it was crawled, and a page you rewrote last year is still in there in its old form. For a brand whose positioning has moved, the archive is a durable record of the previous positioning.

## How do you block CCBot, and should you?

One group, with the token Common Crawl publishes. The user agent it sends is `CCBot/2.0 (https://commoncrawl.org/faq/)`.

robots.txt: opt out of the Common Crawl archive

```
User-agent: CCBot
Disallow: /
```

Whether you should is a values question rather than a marketing one, and the honest answer cuts both ways. Common Crawl is the closest thing the open web has to a public archive, and it is used by researchers who are building no product at all. Blocking CCBot removes you from that too, and it is not retroactive in either direction: snapshots already published stay published.

## When did the snapshots start, and can one be removed?

Common Crawl has been publishing since 2013, and the archive is cumulative rather than a rolling window: FineWeb, built on top of it, draws on 114 separate snapshots running from `CC-MAIN-2013-20` to `CC-MAIN-2025-26`. A page that was public in 2016 is in the archive whether or not the site still exists.

That is the part worth understanding before adding a `Disallow`. Blocking CCBot stops future collection and removes nothing already published, and there is no takedown route through the crawler. It also means a robots.txt change made today reaches no model that has already been trained, which is the same forward-only limit that applies to every training crawler on this list.

## Sources

- [Common Crawl: CCBot](https://commoncrawl.org/ccbot), checked 2026-08-18
- [FineWeb dataset card, built from 114 Common Crawl snapshots](https://huggingface.co/datasets/HuggingFaceFW/fineweb), checked 2026-08-18