GPTBot

Why does GPTBot matter?

GPTBot matters because it is where the training-data decision gets made, and because the decision has a cost either way. Allowing GPTBot means your writing can contribute to the next OpenAI model, which is how a brand becomes something an assistant knows without having to look it up. Disallowing it keeps your content out of that training and changes nothing about whether ChatGPT can find you today. The pressure is not hypothetical: on 1 July 2025 Cloudflare announced it was changing the default to block AI crawlers unless they pay creators, and reported that Anthropic’s crawlers fetched roughly 30,000 pages for every visitor they sent back.

What nobody can tell you is the size of what you are giving up or keeping. OpenAI does not publish how often GPTBot re-crawls a site, how much of any site it has already collected, or whether adding a Disallow line withdraws anything gathered before you added it. Treat the choice as forward-looking only.

How is GPTBot different from OpenAI’s other bots?

GPTBot trains, OAI-SearchBot indexes, and ChatGPT-User fetches on demand. They are three separate robots.txt tokens with three separate jobs, and the single most repeated error in AI crawler advice is treating them as one bot. Blocking GPTBot to “keep out of AI” leaves ChatGPT search untouched. Blocking OAI-SearchBot is what removes a site from ChatGPT search results.

A fourth, OAI-AdsBot, validates advertisement landing pages and says nothing about AI search visibility. The exact user agent GPTBot sends is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and OpenAI publishes its address ranges at openai.com/gptbot.json so a log line claiming to be GPTBot can be checked rather than believed.

Can GPTBot reach your site right now?

Check two things separately, because on a lot of sites they disagree. The first is policy: what your robots.txt asks GPTBot to do. The second is evidence: what your server actually returns when a request arrives carrying GPTBot’s user agent. A site can allow GPTBot in robots.txt and still turn it away at the edge with a 403, a challenge or a payment demand, and that gap is invisible from the robots.txt alone.

Blocking GPTBot is one robots.txt group, and it belongs on its own rather than inside a blanket rule:

User-agent: GPTBot
Disallow: /

# Deliberately absent: OAI-SearchBot, ChatGPT-User.
# Blocking those is what removes a site from ChatGPT.

The recommendation, and it is a position rather than a summary: block GPTBot if you do not want your writing in training data, and leave OAI-SearchBot alone. Most sites that say “block AI” mean the first and get the second by accident. One more thing worth knowing before you decide anything on the strength of a rendered page: Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found GPTBot requesting JavaScript files in 11.5% of its requests and executing none of them. Whatever GPTBot collects, it collects from the server-rendered HTML.

Which tool checks this for you?

Related terms

• OAI-SearchBotOAI-SearchBot is the crawler that builds the index behind ChatGPT search.
• ChatGPT-UserChatGPT-User is the agent that fetches a page because a ChatGPT user asked about it, rather than as part of a crawl.
• ClaudeBotClaudeBot is Anthropic’s training crawler, documented as collecting web content that can contribute to training its models.
• CCBot (Common Crawl)CCBot is the crawler run by Common Crawl, a non-profit that publishes an open archive of the web for anyone to analyse.