# Training Data

> Source: https://rankxai.com/glossary/training-data · Last updated: 2026-08-18

Training data is the body of text a language model learns from before it answers anything. Most public web training corpora are not crawled by the model’s owner directly: they are filtered from Common Crawl’s open archive, which is why a training opt-out has more than one door to close.

## How does a page actually get into training data?

By two routes, and most opt-outs only close one. The direct route is a vendor’s own training crawler, [GPTBot](/glossary/gptbot) or [ClaudeBot](/glossary/claudebot), which honours a robots.txt rule aimed at it. The indirect route is [Common Crawl](/glossary/ccbot), whose open archive is filtered into the datasets model builders actually use: FineWeb, one of the most widely used, draws on 114 Common Crawl snapshots running from 2013 to 2025.

So a robots.txt that names the vendor bots and not CCBot leaves the wider pathway open. It is also the pathway with the least visibility attached: nobody publishes which corpus was built from which snapshot, or which model consumed which corpus.

## Why is a training opt-out only ever forward-looking?

Because a robots.txt rule governs future requests and nothing else. Content already collected stays collected, models already trained stay trained, and archived snapshots stay published. No vendor documents a withdrawal process through the crawler, and none of them publishes what they hold about any given site.

That makes the decision less dramatic than it feels in either direction. Opting out now does not remove you from anything that exists today; leaving the door open does not commit you to anything beyond the next crawl. The choice is about the next model, not this one.

## What does training data mean for AI visibility?

It is the half of an assistant’s answer you cannot reach on any useful timescale. When a model names your brand it may be recalling training or reading a retrieved page, and nothing in the response says which. Retrieval you can influence this week by publishing; training you influence at the pace of model releases, and never precisely.

The recommendation follows from that asymmetry and it is a position rather than a summary. Optimise for retrieval, because it is the half with a feedback loop, and treat any tool claiming to measure or isolate your training-data influence as estimating something nobody can verify. A brand that is well described across the retrievable web will eventually be well described in training too, and the reverse is not a strategy.

One caveat on that, because it cuts against the advice: content behind a login or a paywall is neither retrievable nor trainable, and a company whose best material sits there is invisible to both halves at once. That is a legitimate commercial choice and it should be made deliberately rather than discovered later.

## Sources

- [Common Crawl: CCBot](https://commoncrawl.org/ccbot), checked 2026-08-18
- [FineWeb dataset card, built from 114 Common Crawl snapshots](https://huggingface.co/datasets/HuggingFaceFW/fineweb), checked 2026-08-18
- [OpenAI bot documentation, on disallowing GPTBot](https://developers.openai.com/api/docs/bots), checked 2026-08-18