PerplexityBot

Why is PerplexityBot the easiest allow decision of the seven?

Because Perplexity has removed the ambiguity that makes the other decisions hard. Its documentation says PerplexityBot “is not used to crawl content for AI foundation models”, so the usual trade, visibility now against training later, does not apply. If you want to appear in Perplexity, allow it. If you do not, block it and lose nothing else.

One caveat applies here and to every vendor statement in this glossary: this is Perplexity describing its own practice, and nobody outside Perplexity can verify what the crawled pages are subsequently used for. It is a clear commitment rather than a proof. It is also clearer than any other vendor has offered, and a commitment somebody can be held to is worth more than the silence everywhere else.

The published user agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot), and the addresses are at perplexity.com/perplexitybot.json. Worth reading closely: the vendor’s own string has an unbalanced bracket in it, which will defeat a parser written to expect matching parentheses.

What about Perplexity-User?

Perplexity-User is the other half, and it behaves differently on purpose. It fetches a page because somebody asked Perplexity about it, and the documentation states that “since a user requested the fetch, this fetcher generally ignores robots.txt rules”. That is the most direct statement any vendor makes on the subject, and it is worth reading as written rather than softened.

So the two Perplexity agents want two different mental models. PerplexityBot is a crawler you choose to allow. Perplexity-User is a visitor arriving through an assistant, and stopping it needs a firewall rule rather than a robots.txt line.

Which signal does Perplexity return that the others do not?

Its own search queries. Perplexity is the one assistant that hands back the searches it ran alongside the answer, in a search_queries field, so the sub-queries behind a Perplexity answer are observed rather than inferred. Every other assistant returns the answer and leaves the retrieval invisible.

That makes Perplexity disproportionately useful for research even where it is not a large source of traffic: it is the one place you can read, rather than guess, how a question was decomposed before it was answered. What to do with that is the subject of the query fan-out entry.

Is it actually getting through?

Related terms

• GPTBotGPTBot is OpenAI’s training crawler.
• ClaudeBotClaudeBot is Anthropic’s training crawler, documented as collecting web content that can contribute to training its models.
• ChatGPT-UserChatGPT-User is the agent that fetches a page because a ChatGPT user asked about it, rather than as part of a crawl.
• AI CitationAn AI citation is a link an AI assistant attaches to its answer, naming a page as a source.