# AI Bot Tracking: How to Verify AI Crawler Hits

> Source: https://rankxai.com/blog/track-ai-bots · Last updated: 2026-09-10

AI bot tracking is the practice of identifying, verifying and counting visits from AI crawlers such as GPTBot, ClaudeBot and OAI-SearchBot in your server or edge logs. JavaScript analytics cannot see these bots, so the record lives in request logs, and a user-agent string alone is not proof of identity.

## What is AI bot tracking?

AI bot tracking is the practice of finding and verifying AI crawler visits in your own request logs, and the verifying is what separates it from ordinary bot reporting. The crawlers are named agents run by OpenAI, Anthropic, Perplexity, Google, Apple, Amazon and others, and each announces itself with a documented product token in the user-agent header. That token is a claim, not a credential.

The question AI bot tracking answers is narrow and worth stating plainly. An assistant can only cite a page it has read, so before asking why a brand is missing from AI answers, it is worth establishing whether the crawlers have arrived at all. A site nobody crawls has a technical problem rather than a content one.

AI bot tracking is one input to a larger measurement, not the whole of it. Crawl activity tells you a page was fetched. Whether that page was then retrieved, cited or named in an answer is a separate reading, and [the measurement framework in full](/blog/measure-ai-search-visibility) sets out how the two fit together.

## Where AI bot activity shows up, and where it does not

AI bot activity appears in server and CDN request logs, and almost nowhere else. Every AI crawler visit is an HTTP request, so it lands in the access log of whatever serves the response: your origin, your CDN, or both.

Google Analytics 4 does not record these visits, and the reason is structural rather than a configuration mistake. GA4 fires when a JavaScript tag executes in a rendered page. AI crawlers fetch the HTML and leave without running scripts, so the tag never executes and no event is ever sent. No analytics platform built on a client-side tag can see these AI systems arrive.

### The measurement behind the no-JavaScript claim, and its date

The evidence that AI crawlers skip JavaScript comes from one study, and most pages repeating it leave the date off. [Vercel and MERJ analysed their network traffic](https://vercel.com/blog/the-rise-of-the-ai-crawler) and published on 17 December 2024: 569 million GPTBot fetches, 370 million from Claude and 24.4 million from PerplexityBot. GPTBot requested JavaScript files on 11.50 per cent of requests and ClaudeBot on 23.84 per cent, and neither executed them.

That study is now twenty-one months old, which matters for a claim about live systems. No AI vendor has documented adding JavaScript rendering to a crawler since, and no fresh primary replication is public, so the finding stands as the best available evidence rather than as a current measurement. Gemini is the documented exception, because it reaches pages through Googlebot's rendering infrastructure.

The practical consequence for a site is worth separating from the tracking question. Content that only exists after a client-side script runs is absent from what these crawlers read, which is a visibility problem in its own right, covered in [what AI crawlers actually read](/blog/how-ai-crawlers-read-your-site).

There is a second limit worth naming, because it is the one people run into after the logs are working. A log line proves that a bot arrived and got a response. It says nothing about what the engine did next, and for Googlebot the answer is readable somewhere else entirely: the index verdict for that URL, which can move from indexed to not indexed in a week while the crawl log stays perfectly healthy. [How to check when Google indexed a page](/blog/check-when-google-indexed-page) covers that reading and the ten pages of this site that made the move.

## Which AI bots to look for in your logs

Look for the documented product tokens, and group them by what each bot is for, because the three purposes carry different consequences across the AI platforms. A training crawler collects pages that may train a future AI model, which is how most LLMs acquired their generative ability in the first place. A search crawler builds the index an assistant answers from. A user-fetch agent retrieves one page because a person asked about it.

| Token | Vendor | Purpose | Honours robots.txt |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search index | Yes |
| ChatGPT-User | OpenAI | User fetch | May ignore |
| ClaudeBot | Anthropic | Training | Yes |
| Claude-SearchBot | Anthropic | Search index | Yes |
| Claude-User | Anthropic | User fetch | Yes |
| PerplexityBot | Perplexity | Search index | Yes |
| Perplexity-User | Perplexity | User fetch | May ignore |
| Amazonbot | Amazon | Training | Yes |
| Applebot | Apple | Search index | Yes |
| CCBot | Common Crawl | Training | Yes |
| meta-externalagent | Meta | Training | Yes |
| Bytespider | ByteDance | Training | No official statement |

Two tokens in wide circulation never appear in a log file at all. Google-Extended and Applebot-Extended are control tokens: they exist so a site can express a training preference in robots.txt, and nothing fetches under either name. Counting zero visits from Google-Extended is the expected result rather than a sign that something is broken.

Googlebot belongs on your list too, and its absence from AI crawler roundups catches people out. Google AI Overviews and AI Mode are served from Google's own Search index, so no separate AI crawler fetches your pages for them. Google's documentation states that AI is built into Search, and that robots.txt directives for Googlebot are therefore the control for how sites are crawled for Search. Google-Extended limits AI training and grounding in Google's other systems.

The consequence for your log reading is that one crawler serves two purposes. Ordinary SEO crawling and AI Overviews arrive on the same Googlebot user-agent, so unlike the other search engines and assistants in the table, you cannot separate the AI visit from the classic one in the log at all. Everything Googlebot fetches feeds both.

Search crawlers deserve attention first if visibility is the goal. Blocking OAI-SearchBot removes a site from ChatGPT's search results and blocking PerplexityBot removes it from Perplexity's index, while blocking a training crawler changes nothing about whether an assistant can cite the site today. You can [check whether crawlers can reach you](/tools/ai-crawler-access-checker) without reading a log at all.

## How to verify an AI bot is genuine

Verify an AI bot by its source IP address, because the user-agent is a header the client chooses. Anyone can send a request announcing itself as GPTBot, and unverified user-agent counts are the most common defect in AI bot tracking: they inflate the numbers with scrapers and security scanners wearing a borrowed name.

Two verification methods are documented across the industry. Most AI vendors publish a JSON file of the IP ranges their crawlers use, and you check the request's source address against it. Google additionally documents a reverse DNS check, described in [Google's own verification guide](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot): resolve the IP to a hostname, confirm it ends in a Google domain, then resolve that hostname forward and confirm it returns the same address.

OpenAI documents the first method only. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) gives a separate JSON endpoint for each of GPTBot, OAI-SearchBot and ChatGPT-User, and offers no reverse DNS alternative, so the published list is the whole verification path for those three.

## Which AI vendors publish a usable IP list

Six AI vendors publish machine-readable IP lists across ten endpoints, and they differ enough that the advice to verify against them needs qualifying. We fetched every documented endpoint in our crawler roster on 6 September 2026 and read the files, counting prefixes by address family and recording each file's own timestamp.

| Bot | Ranges published | Includes IPv6 | File last stamped |
| --- | --- | --- | --- |
| GPTBot | 21 | No | 30 Oct 2025 |
| OAI-SearchBot | 35 | No | 2 Jan 2026 |
| ChatGPT-User | 207 | No | 4 Sep 2026 |
| Claude (all three bots) | 26 | No | 18 Aug 2026 |
| PerplexityBot | 8 | No | 7 Feb 2025 |
| Perplexity-User | 4 | No | 17 Oct 2025 |
| Applebot | 33 | No | 31 Jul 2026 |
| MistralAI-Index | 2 | No | 19 Apr 2026 |
| MistralAI-User | 4 | No | 19 Feb 2025 |
| DuckAssistBot | 486 | No | 1 Sep 2026 |
| Googlebot | 317 | Yes, 147 of them | 4 Sep 2026 |

### Every AI vendor list except Google's is IPv4 only

All ten AI vendor endpoints published IPv4 ranges and nothing else. Google was the only publisher in the set carrying IPv6: 147 of Googlebot's 317 prefixes, and 136 of the 272 in its special-crawlers file. The standard advice to verify against the published list does not mention it, and the consequence is direct.

If your origin answers over IPv6 and a request arrives claiming to be GPTBot from an IPv6 address, the published list cannot settle it either way. No vendor states whether its list is exhaustive, and OpenAI's documentation does not mention IPv6 at all.

So the request may be a real crawler on an unlisted address, or a forgery the list would have caught over IPv4. Record those as a third outcome, separate from verified and rejected: filing them as fake undercounts real crawler activity, and filing them as genuine trusts the header you set out to check.

### The lists age at very different rates

Freshness across the ten endpoints spans nineteen months. PerplexityBot's file was stamped 7 February 2025 and MistralAI-User's 19 February 2025, both more than eighteen months before we read them. ChatGPT-User's was stamped 4 September 2026, two days before. A stale list produces false negatives, so a verification failure against one of the older files is weaker evidence of spoofing than a failure against a recent one.

The size range is just as wide, from 2 prefixes for MistralAI-Index to 486 for DuckAssistBot and 1,058 in Google's user-triggered fetchers file. OpenAI's own split is striking, with 207 ranges for ChatGPT-User against 21 for GPTBot. Perplexity runs the other way round, publishing 4 ranges for its user agent and 8 for its crawler, so across six vendors this is a spread rather than a rule.

## Two limits of AI bot IP verification

Two limits constrain IP verification whatever tooling you build on it, and both are properties of what the vendors publish rather than of your setup.

Anthropic publishes one list for all three Claude bots. The file at claude.com/crawling/bots.json holds 26 bare IPv4 prefixes with no per-bot labels, so an address can confirm that a request came from Anthropic and cannot tell ClaudeBot apart from Claude-SearchBot or Claude-User. Purpose-level reporting for Claude has to come from the user-agent string, which is the field you were verifying in the first place.

Amazon documents three URLs for its bots and all three serve HTML pages rather than JSON. They are readable by a person and not by the same parser that handles the other ten, which is why Amazon sits outside the table above. ByteDance and xAI publish no verification data at all, so Bytespider and Grok traffic cannot be verified by either documented method.

## How to set up AI bot tracking

Set up AI bot tracking in four steps, none of which needs an analytics platform. The work is log collection, filtering, verification and storage, and where each step happens depends on whether your CDN or your origin sees the request first.

1. **Get the logs.** Most CDNs expose request logs through an API or a scheduled export, with the user-agent and client IP intact. On a server-side stack with no CDN in front of it, the origin access log already holds both fields and needs nothing configured.
2. **Filter for specific bots** by user-agent token rather than a generic bot pattern, because a broad match sweeps in uptime monitors and security scanners and inflates every count that follows. On a plain access log that is one command: `grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot|CCBot|meta-externalagent|Bytespider' access.log`
3. **Verify each candidate** against the vendor's published IP ranges. Every one of these files is a JSON object holding a `prefixes` array of `ipv4Prefix` strings and a `creationTime`, so the check is a CIDR match against the source address and about twenty lines of code. Cache the lists rather than fetching them per request, and refresh them on a schedule, because the files change without notice and none of them announces it.
4. **Store one of three verdicts, not two.** `verified` when the address matches the list, `rejected` when it does not, and `unresolved` when the request came in over IPv6 and no published list can speak to it either way. Keep the bot's purpose beside the verdict. Aggregating verified bot visits later is cheap, and recovering them from raw logs a month afterwards is not.

Log analysers will do the first two steps for you, with one caveat worth knowing. Screaming Frog's Log File Analyser imports a log and offers a Verify Bots option that checks hits against publicly confirmed IP lists, and GoAccess will parse and group a log from the command line. Screaming Frog's documentation does not say which lists it checks or how fresh they are, and after the table above you now know why that matters.

Keep your robots.txt file separate from the measurement in your head. The robots.txt file states what you permit, and the log states what happened. When the two disagree the log wins, because a CDN rule or a WAF can refuse an AI crawler that your robots.txt file explicitly allows, and nothing in the file will tell you it happened.

## What AI crawler numbers do and do not tell you

AI crawler counts tell you that pages were fetched, and they are routinely asked to carry more weight than that. The most quoted figure in this space is Cloudflare's crawl-to-refer ratio, which divides requests from a platform's crawlers by the visits that platform referred back. It is a useful shape and it comes with a caveat its popular retellings drop.

[Cloudflare's own post](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/), published 1 July 2025, counts referrals using the Referer header and states that traffic referred by Claude's native app carries no such header. Cloudflare says directly that the calculations may overstate the ratios and that it is unclear by how much. Anthropic's figure was 70,900 to 1 for the week of 19 to 26 June 2025.

Those ratios also move by an order of magnitude between windows, which is a reason to read any single one carefully rather than a reason to distrust the metric. Aggregate numbers describe the web as a whole, and your own crawler traffic is what governs your decisions. Measuring it is the argument for tracking bots on your own logs at all.

Attribution is the harder half, and worth saying plainly. Traditional search results hand you a referrer and a query in Search Console, while an assistant that reads your page and answers from it leaves no trace on your site at all. Crawl logs are the only record that the read happened, which is why they carry weight out of proportion to their detail.

### One denominator trap in the published crawl statistics

Cloudflare has published two training-share figures that are not comparable, and they are already circulating as though they were one series. Its August 2025 analysis put training at nearly 80 per cent of AI bot crawling. Its report of 1 July 2026 put AI training crawlers at 52 per cent of crawler requests, up from 22 per cent in spring 2025.

Read as a trend, that looks like a fall from 80 to 52. It is not one. The 2026 figure is rising against its own stated baseline of 22 per cent, and the two denominators are different, with the later report not stating its own explicitly. The safe conclusion is that these two numbers cannot be subtracted, and anything built on the gap between them is built on nothing.

## What to do with AI bot tracking data

Turn AI bot tracking data into three readings, each of which supports a different decision. Start from verified requests only, grouped by bot and by purpose, over a window long enough to smooth out crawl bursts.

1. **Coverage.** Which of your pages have search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot actually fetched? Pages that earn nothing from AI search because they were never read are a different problem from pages that were read and not cited.
2. **Response codes.** A crawler receiving 404s, 403s or timeouts is being turned away by your own infrastructure. This is the most common finding worth acting on and the cheapest to fix.
3. **Purpose mix.** Training crawlers dominate most logs and cannot send you a visitor. Separating them keeps a large number from reading as visibility it is not.

Do not read a change in crawl volume as a change in visibility. Crawl frequency responds to publishing cadence, sitemap changes and the vendors' own scheduling, none of which is under your control or tied to whether an assistant cites you. Being crawled is necessary for citation and it does not produce citation, so keep the two measurements apart.

Blocking decisions follow from the purpose column rather than from volume. A training crawler that costs bandwidth and returns nothing is a defensible thing to disallow. A search crawler is the mechanism by which an assistant can cite you at all, and blocking it to reduce load removes the visibility you were trying to measure.

If a search crawler is missing from the log entirely, the next place to look is robots.txt rather than the tracking setup. On WordPress that check has its own failure modes, and [what the WordPress AI block lists actually disallow](/blog/wordpress-robots-txt-ai) walks through them.

## Tracking AI bot activity with RankX AI

RankX AI is our own platform, and this section is the only part of this article selling it. RankX AI checks crawler access from the outside, reading your robots.txt against the current roster of AI crawler tokens. It reports which of the thirteen bots above are allowed, which are blocked and which rules are ambiguous. That answers the access half of the question without needing your logs at all.

For the visibility half, RankX AI runs prompt sets against the assistants and records whether your brand is named and which sources were cited. That pairs with crawl data in the way this article describes: access explains whether a page could be read, and prompt tracking measures whether it is being used. Our own numbers for both are published in [our AI visibility baseline](/blog/our-ai-visibility-baseline), and you can [see plans](/pricing).

The log-side work described above needs no product at all. Any CDN or server access log plus the ten published IP files gives you a verified AI bot report, and the files are public. That is the part of AI bot tracking worth building first, whatever you use to measure the rest.

## How do you know if an AI bot has visited your site?

Read your server or CDN request logs and filter the user-agent field for the documented product tokens. OpenAI sends GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic sends three Claude agents, Perplexity sends two, and the rest of the roster covers Amazonbot, Applebot, CCBot, meta-externalagent and Bytespider. JavaScript analytics such as Google Analytics 4 cannot record any of these visits, because the bots do not execute JavaScript, so the tag never fires. Anything you learn about AI crawler activity comes from the log line, not from an analytics dashboard.

## Can you tell a real GPTBot from a fake one?

Only by checking the source IP address, because the user-agent string is a header any client can set. OpenAI publishes the ranges GPTBot crawls from at openai.com/gptbot.json, and a request claiming to be GPTBot from an address outside that list is not GPTBot. One caveat measured on 6 September 2026: that file held 21 IPv4 ranges and no IPv6 ranges at all, and was last stamped 30 October 2025. If your origin is reachable over IPv6, a request over IPv6 cannot be confirmed against it either way.

## Which AI vendors publish IP ranges you can verify against?

Six of them, on ten machine-readable endpoints, as measured on 6 September 2026: OpenAI for GPTBot, OAI-SearchBot and ChatGPT-User separately, Anthropic for all three Claude bots on one shared list, Perplexity for PerplexityBot and Perplexity-User, Apple for Applebot, Mistral for MistralAI-Index and MistralAI-User, and DuckDuckGo for DuckAssistBot. Google publishes its own set covering Googlebot and its special crawlers. Amazon documents three URLs that serve HTML pages rather than JSON. ByteDance and xAI publish nothing.

## Does Google Analytics track AI crawlers?

No. Google Analytics 4 records a visit when a JavaScript tag runs in the page, and AI crawlers do not run JavaScript. The Vercel and MERJ analysis published on 17 December 2024 found GPTBot fetched JavaScript files on 11.5 per cent of its requests and ClaudeBot on 23.84 per cent, and that neither executed them. GA4 can show you human referral traffic arriving from an assistant, which is a different measurement from the crawler that read the page weeks earlier.

## How often do AI crawlers visit a site?

It varies far too much between sites for a published average to be useful, which is the reason to measure your own. Cloudflare's aggregate crawl-to-refer ratio for Anthropic has been quoted at 70,900 to 1 for late June 2025 and at figures an order of magnitude lower during 2026, depending on the window and the denominator chosen. Cloudflare also states that referrals from native apps carry no Referer header, so its own ratios may overstate by an amount it describes as unclear.

## Should you block AI crawlers once you can see them?

That depends on which bot, and the purposes are worth separating before you decide. Blocking OAI-SearchBot removes a site from ChatGPT search results, and blocking PerplexityBot removes it from Perplexity's index, so both cost visibility directly. Blocking a training crawler such as GPTBot or CCBot does not affect whether an assistant can cite you today. Control tokens such as Google-Extended and Applebot-Extended never fetch anything, so a robots.txt rule is the whole mechanism for those two.