Training Data

How does a page actually get into training data?

By two routes, and most opt-outs only close one. The direct route is a vendor’s own training crawler, GPTBot or ClaudeBot, which honours a robots.txt rule aimed at it. The indirect route is Common Crawl, whose open archive is filtered into the datasets model builders actually use: FineWeb, one of the most widely used, draws on 114 Common Crawl snapshots running from 2013 to 2025.

So a robots.txt that names the vendor bots and not CCBot leaves the wider pathway open. It is also the pathway with the least visibility attached: nobody publishes which corpus was built from which snapshot, or which model consumed which corpus.

Why is a training opt-out only ever forward-looking?

Because a robots.txt rule governs future requests and nothing else. Content already collected stays collected, models already trained stay trained, and archived snapshots stay published. No vendor documents a withdrawal process through the crawler, and none of them publishes what they hold about any given site.

That makes the decision less dramatic than it feels in either direction. Opting out now does not remove you from anything that exists today; leaving the door open does not commit you to anything beyond the next crawl. The choice is about the next model, not this one.

What does training data mean for AI visibility?

It is the half of an assistant’s answer you cannot reach on any useful timescale. When a model names your brand it may be recalling training or reading a retrieved page, and nothing in the response says which. Retrieval you can influence this week by publishing; training you influence at the pace of model releases, and never precisely.

The recommendation follows from that asymmetry and it is a position rather than a summary. Optimise for retrieval, because it is the half with a feedback loop, and treat any tool claiming to measure or isolate your training-data influence as estimating something nobody can verify. A brand that is well described across the retrievable web will eventually be well described in training too, and the reverse is not a strategy.

One caveat on that, because it cuts against the advice: content behind a login or a paywall is neither retrievable nor trainable, and a company whose best material sits there is invisible to both halves at once. That is a legitimate commercial choice and it should be made deliberately rather than discovered later.

Related terms

• CCBot (Common Crawl)CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web for anyone to analyse.
• GPTBotGPTBot is OpenAI’s training crawler.
• Knowledge CutoffA knowledge cutoff is the date after which a model’s training data contains nothing.
• AI CrawlerAn AI crawler is an automated fetcher run by an AI company to collect web pages.