# GPTBot

> Source: https://rankxai.com/glossary/gptbot · Last updated: 2026-08-18

GPTBot is OpenAI’s training crawler. OpenAI documents it as collecting web content to train its foundation models, and it is one of four OpenAI bots with separate jobs. GPTBot does not fetch pages to answer live questions, so disallowing it in robots.txt does not remove a site from ChatGPT search results.

## Why does GPTBot matter?

GPTBot matters because it is where the training-data decision gets made, and because the decision has a cost either way. Allowing GPTBot means your writing can contribute to the next OpenAI model, which is how a brand becomes something an assistant knows without having to look it up. Disallowing it keeps your content out of that training and changes nothing about whether ChatGPT can find you today. The pressure is not hypothetical: on 1 July 2025 Cloudflare announced it was [changing the default to block AI crawlers unless they pay creators](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), and reported that Anthropic’s crawlers fetched roughly 30,000 pages for every visitor they sent back.

What nobody can tell you is the size of what you are giving up or keeping. OpenAI does not publish how often GPTBot re-crawls a site, how much of any site it has already collected, or whether adding a `Disallow` line withdraws anything gathered before you added it. Treat the choice as forward-looking only.

## How is GPTBot different from OpenAI’s other bots?

GPTBot trains, OAI-SearchBot indexes, and ChatGPT-User fetches on demand. They are three separate robots.txt tokens with three separate jobs, and the single most repeated error in AI crawler advice is treating them as one bot. Blocking GPTBot to “keep out of AI” leaves ChatGPT search untouched. Blocking OAI-SearchBot is what removes a site from ChatGPT search results.

OpenAI’s three website-facing crawlers, from OpenAI’s bot documentation, read 18 August 2026.

- Bot: `GPTBot`. What it is for: Training OpenAI’s foundation models. What disallowing it does: Keeps future training out. No effect on ChatGPT search.
- Bot: `OAI-SearchBot`. What it is for: Indexing pages for ChatGPT search. What disallowing it does: Removes the site from ChatGPT search results.
- Bot: `ChatGPT-User`. What it is for: Fetching a page because a user asked about it. What disallowing it does: Little. OpenAI states robots.txt rules “may not apply”, because a human requested the fetch.

A fourth, `OAI-AdsBot`, validates advertisement landing pages and says nothing about AI search visibility. The exact user agent GPTBot sends is `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`, and OpenAI publishes its address ranges at `openai.com/gptbot.json` so a log line claiming to be GPTBot can be checked rather than believed.

## Can GPTBot reach your site right now?

Check two things separately, because on a lot of sites they disagree. The first is policy: what your robots.txt asks GPTBot to do. The second is evidence: what your server actually returns when a request arrives carrying GPTBot’s user agent. A site can allow GPTBot in robots.txt and still turn it away at the edge with a 403, a challenge or a payment demand, and that gap is invisible from the robots.txt alone.

Blocking GPTBot is one robots.txt group, and it belongs on its own rather than inside a blanket rule:

robots.txt: block training, keep AI search visibility

```
User-agent: GPTBot
Disallow: /

# Deliberately absent: OAI-SearchBot, ChatGPT-User.
# Blocking those is what removes a site from ChatGPT.
```

The recommendation, and it is a position rather than a summary: block GPTBot if you do not want your writing in training data, and leave OAI-SearchBot alone. Most sites that say “block AI” mean the first and get the second by accident. One more thing worth knowing before you decide anything on the strength of a rendered page: Vercel’s analysis of AI crawler traffic across its network, published 17 December 2024, found GPTBot requesting JavaScript files in 11.5% of its requests and executing none of them. Whatever GPTBot collects, it collects from the server-rendered HTML.

## Sources

- [OpenAI bot documentation](https://developers.openai.com/api/docs/bots), checked 2026-08-18
- [Cloudflare: Content Independence Day, 1 July 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), checked 2026-08-18
- [Vercel: the rise of the AI crawler, 17 December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), checked 2026-08-18