OAI-SearchBot

What does OAI-SearchBot do that GPTBot does not?

OAI-SearchBot builds a searchable index; GPTBot builds a training corpus. That is the whole distinction, and it decides which of the two you can afford to block. Content OAI-SearchBot has indexed can be surfaced and linked in a ChatGPT answer today. Content GPTBot has collected may influence a model released in a year, with no link and no attribution.

OpenAI publishes the user agent as Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Note that it opens with an ordinary Chrome string: a filter written to match “bot” or “compatible” at the start of a user agent will miss it entirely.

Why do sites block OAI-SearchBot by accident?

Because the instruction people mean to give is “keep my content out of AI training”, and the rule they write is a blanket one. Two patterns do it. The first is a wildcard group that disallows everything and then allows a named list written before ChatGPT search existed. The second is a content delivery network setting labelled “block AI bots”, which is one switch over a category the vendor defines rather than a per-purpose choice you made.

The result is a site absent from ChatGPT search results whose owner believes it has only opted out of training. It is worth checking rather than assuming, because nothing reports it: there is no console, no warning, and no traffic to notice missing, since a citation that never happened leaves no trace.

How do you confirm a request really came from OAI-SearchBot?

By address, never by user agent. A user agent string is a request header, so anything can send OAI-SearchBot/1.4 and plenty of things do, including scrapers hoping a permissive rule applies to them. OpenAI publishes the address ranges its crawlers use as JSON, one file per bot:

• openai.com/searchbot.json for OAI-SearchBot
• openai.com/gptbot.json for GPTBot
• openai.com/chatgpt-user.json for ChatGPT-User

Match the source address of the request against the file for the bot it claims to be. Anything that fails is not OpenAI, whatever its headers say. The honest limit: this tells you a request was genuine, and it tells you nothing about whether the page was used, because OpenAI publishes no feedback of any kind about what its index retained. Nothing in this verification path tells you frequency either, so a site seeing one OAI-SearchBot request a month and a site seeing a thousand have no way to know which of them is normal.

Related terms

• GPTBotGPTBot is OpenAI’s training crawler.
• ChatGPT-UserChatGPT-User is the agent that fetches a page because a ChatGPT user asked about it, rather than as part of a crawl.
• AI CitationAn AI citation is a link an AI assistant attaches to its answer, naming a page as a source.
• Answer EngineAn answer engine is a search product that returns a written answer rather than a ranked list of links, citing a handful of sources inside it.