We use essential storage to run the site. With your permission we also use analytics and advertising cookies to see which pages help and which adverts brought you here. See the cookie policy.
The AI Crawler Access Checkershows which AI bots you block.
An AI crawler access checker shows whether AI bots such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your website. RankX AI tests your robots.txt and your live server responses for all 27 documented AI bots, free and with no account.
A free AI crawler checker. No account. We fetch only your homepage and robots.txt, once per bot, and cache the result for 30 minutes.
Example resultSample data
halewick.examplerobots.txt: found
Crawler
Purpose
Verdict
GPTBot
Training
Open
OAI-SearchBot
AI search
Open
ClaudeBot
Training
Robots disallows
Claude-SearchBot
AI search
CDN blocks it
PerplexityBot
AI search
Open
What it is
What is an AI crawler access checker?
An AI crawler access checker is a free tool that shows whether AI bots can access your website and crawl your site. The AI Crawler Access Checker reads your robots.txt for every documented AI bot, then asks your server as each one, so you see both answers.
What it is
Two answers for each AI bot, side by side
For each of the 27 documented AI crawlers, from GPTBot and OAI-SearchBot to ClaudeBot and PerplexityBot, the tool states what your robots.txt asks and what your homepage returned when that bot's user agent arrived. A generic robots.txt tester stops at the file.
A robots.txt rule is a request; the server response is what the bot actually got.
Who it is for
Anyone deciding which AI bots to let in
Site owners asking why ChatGPT or Perplexity never cites them, SEOs auditing a client's CDN and bot rules, and developers checking a robots.txt change before it ships. Run it before you decide which AI crawlers to allow or block, and again after any CDN, WAF or robots.txt change that could stop you appearing in AI answers.
The person who changed the firewall is rarely the person who reads the AI visibility report.
What it costs to run
Free, with no account, one domain at a time
A check is free: it reads one domain's robots.txt and requests its homepage once per bot it can probe, 24 of them, plus two controls. Results are cached for 30 minutes per domain and keep verdicts only. For a daily re-check, RankX AI re-reads robots.txt and your homepage on every account and reports when access gets worse.
One homepage, one moment: deeper pages and bots verified by IP address are out of its sight.
What this checks
How does the AI crawler check work?
The AI Crawler Access Checker reads what your robots.txt asks and what your server actually returns when a crawler arrives. Policy and evidence are read separately, because they often disagree. Use it to check if AI crawlers can access your site and see which bots are allowed or blocked.
01 / Policy
robots.txt, parsed under RFC 9309
The tool parses your robots.txt under RFC 9309, the rule set crawlers themselves apply: groups merge across the file, the most specific user agent wins, and the longest matching path rule decides. Those three rules are where a naive checker goes wrong: one that stops at the first group it matches, or takes the shortest rule, reports a bot as blocked that the crawler itself reads as allowed.
A checker that reads the file loosely reports blocks you do not have.
02 / Evidence
A live fetch per bot, plus two controls
The checker then requests your homepage once per documented bot with that crawler's own User-Agent header, plus two controls: a normal browser and a plain curl. Browser in and bot out means the edge blocks that user agent. Everything non-browser out means generic bot protection, and the result says so instead of guessing per bot.
Browser in and bot out is your edge deciding, not your robots.txt.
03 / Verdict
Unverifiable is a verdict, not a gap
Each crawler gets one verdict built from both columns. Where nothing can be verified, the verdict says unverifiable: xAI documents no crawler token at all, and ByteDance publishes nothing for Bytespider, so an honest tool refuses to print a green tick for either.
Two vendors on this roster publish nothing, and no tool can honestly say otherwise.
Access is not the same as crawlability
A crawler that gets a 200 still has to find your words in the HTML. The LLM crawlers behind the assistants execute no JavaScript, and only Googlebot and Applebot render, so a page built client-side is accessible and unreadable at the same time: the bot is allowed in and leaves with nothing.
Accessibility and AI crawlability are two separate tests, and the tools built for traditional search engines miss both, because they check what search engines index rather than what each AI bot is served. If every crawler is open here, the next question is whether AI systems can read your content for AI answers without JavaScript, which is what the AI Readiness Score measures.
The 27 AI web crawlers checked, across 12 vendors, from each vendor's own documentation
8TrainingCollects pages for model training corpora.
9AI searchIndexes pages so an assistant can answer from them and cite them.
8User fetchFetches one page because a person asked about it.
2Control tokenNever fetches. The robots.txt token is the whole mechanism.
Googlebot crawls for Google Search, which includes AI Overviews: there is no separate AI Overviews crawler.
Google-Extended
Control token
Is the control
Google-Extended is a robots.txt control, not a crawler: it decides whether Google may use your content to train Gemini models, including the models behind AI Overviews and AI Mode, and to ground Gemini apps. It does not affect inclusion or ranking in Google Search.
Google-Agent
User fetch
May ignore
Google-Agent fetches and navigates pages for agents running on Google infrastructure, acting on a user’s request.
Google-GeminiNotebook
User fetch
May ignore
Google-GeminiNotebook fetches the individual URLs a Gemini Notebook user adds as sources.
DuckAssistBot fetches pages in real time for DuckDuckGo’s AI-assisted answers and is not used to train AI models.
ByteDance / no published documentation
Bytespider
Training
Unverifiable
Bytespider is ByteDance’s crawler, widely observed feeding AI training. ByteDance publishes no documentation for it, so compliance is unverifiable.
xAI / no published documentation
xAI / Grok
Training
Unverifiable
xAI documents no crawler token at all: robots.txt cannot target Grok’s data collection, and compliance is unverifiable.
Roster verified against vendor documentation on 13 September 2026. Each vendor heading links to that vendor's primary source. UA rosters move; the roster is re-verified quarterly.
Why it matters
Why can a CDN block AI crawlers your robots.txt allows?
Since 15 September 2026, Cloudflare offers each new domain an AI crawler preset chosen by whether it runs ads, and AI search crawlers stay allowed in both.
Choosing Block for AI training now blocks Googlebot with it.
Vendor behaviour
Blocking is now per purpose
Every major vendor runs separate bots for training, search and user-requested fetches: blocking GPTBot keeps your content out of OpenAI training and does not touch ChatGPT search, which runs on OAI-SearchBot. Cloudflare said in September 2026 that 17% of the sites on its network block AI training in some way, while fewer than 1% block search bots. Each purpose shows separately here, so you can see which AI crawlers a rule of yours allows and which it blocks.
A blanket AI block removes you from the answers your buyers read, not just from training sets.
Common mistake
The Google trap catches careful people
Google AI Overviews have no dedicated crawler: they are built from ordinary Googlebot crawling, and Google-Extended governs Gemini training and grounding, not inclusion or ranking in Google Search. Yet robots.txt files across the web block Google-Extended believing it opts them out of AI Overviews. The control that does is Search Console's Search generative AI setting, which has applied to every site since 31 August 2026.
The most popular AI-blocking recipe on the web does not do what its comments say it does.
Diagnosis
Silent blocks read as strategy failures
When your site is not appearing in AI answers, the usual diagnosis is content or authority, but sometimes the truth is a 403: the bot asked and your edge refused. Your analytics cannot show it, because a refused bot never reaches the page. A CDN setting you never chose can do this, and so can a bot rule a colleague added for scrapers.
You cannot fix a block you have never seen; checking takes twenty seconds.
What changed at Cloudflare on 15 September 2026
SearchAllowed in both presets
TrainingDisallow AI Training, if the site runs ads
AgentBlocked on ad pages, if the site runs ads
Cloudflare's recommended preset for a new domain from 15 September 2026. A site with no ads is offered Allow on all three.
Cloudflare sorts AI traffic into three behaviours you set separately, Search, Agent and Training: allow, block only on pages that display ads, or block everywhere. Training has a fourth option, Disallow AI Training, which publishes a no-training rule in robots.txt, blocks training-only crawlers such as GPTBot and ClaudeBot, and keeps mixed-use crawlers fetching for search. From 15 September 2026 a new domain is offered one of two presets: with no ads, all three are allowed; with ads, Search is allowed, Training is set to Disallow AI Training and Agent is blocked on pages with ads.
The sharp edge is elsewhere: from the same date, choosing Block for AI training stops Googlebot, Applebot and Bingbot too, because Cloudflare names them as crawlers that both search and train. Existing training blocks, including the one-click Block AI bots from July 2025, were migrated to a new setting, Disallow AI Training, which keeps those three crawling for search.
A robots.txt that allows Googlebot does not override a Block, and a live fetch shows the difference. Bing is the exception to the new setting: Microsoft's support for a robots.txt no-training rule is targeted for early 2027, so until then Disallow AI Training sends Bing nothing.
Choosing Block for AI training at the CDN is now a decision about Google Search, not only about model training.
How to fix it
How do I block AI training but keep AI search open?
Built from the same roster, these crawler rules name current tokens. Copy what fits your policy, then re-run the check: a CDN rule can override robots.txt.
On WordPress there is one more step, because a physical robots.txt file silently disables every rule your SEO plugin generates: blocking AI crawlers in WordPress covers what to check before pasting.
Block AI training, keep AI search
The most common deliberate policy: training crawlers and the two training control tokens are disallowed, and the search bots are simply not mentioned, which leaves them allowed.
robots.txtBlock training
# Block AI training crawlers and training-use control tokens.
# AI search bots are deliberately NOT listed here, so answer
# engines can still cite this site.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
State AI search access explicitly
Allow is already the default, so a group that allows AI crawlers changes nothing by itself. It documents intent, and it protects the search bots from a broader Disallow rule pasted above it later.
robots.txtAllow AI search
# Explicitly allow the AI search and answer-engine crawlers.
# Allow is the default, so this group documents intent and
# protects these bots from a broader Disallow above it.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Applebot
Allow: /
User-agent: MistralAI-Index
Allow: /
User-agent: DuckAssistBot
Allow: /
When robots.txt is not where the block is
If the result shows the CDN blocking bots your robots.txt allows, the fix is in your CDN dashboard, not in robots.txt. On Cloudflare, open Security Settings and then Configure AI bot policies, where Search, Agent and Training are set separately, and leave Search allowed if you want AI search engines citing you.
To refuse AI training without losing Googlebot, set Training to Disallow AI Training rather than Block. Check Bot Fight Mode as well, because it turns away automation whatever category the crawler falls into.
Keep going
Free AI SEO tools for the rest of your AI search visibility
Being accessible to AI bots is the first condition for AI search visibility. These tools cover page readiness, the llms.txt file and whether AI answers name and cite you.
One check answers for one moment. In RankX AI, AI Readiness re-reads your robots.txt, your llms.txt and your homepage every day and reports when an AI crawler's access gets worse, and the SEO audit tool reports that AI crawler access beside its own score, never blended into it. AI visibility tracking then shows whether the AI assistants actually name and cite you once their crawlers can get in.
Questions
AI crawler access checker questions, answered
Direct answers, with the vendor documentation named, including the two bots nobody can verify.
What is the AI Crawler Access Checker, and what does it test?
The AI Crawler Access Checker is a free robots.txt checker for AI crawlers. For every documented AI bot, from GPTBot to ClaudeBot, it reports what your robots.txt asks, parsed under RFC 9309, and what your server actually returns to that bot’s user agent. It keeps policy and evidence apart because on many sites they disagree.
Why show policy and evidence as separate columns?
Because robots.txt is a request, not a lock. Your robots.txt can allow GPTBot while your CDN, WAF or bot rules turn the same bot away with a 403, a challenge or a 402 payment demand, and nothing on your side logs a complaint. That gap is a common surprise, and only a live request can reveal it.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI’s training crawler; ChatGPT search results come from OAI-SearchBot, and pages a ChatGPT user asks about are fetched by ChatGPT-User. Blocking GPTBot keeps your content out of future training and leaves you visible in ChatGPT search, while blocking OAI-SearchBot removes you from it.
Does Google-Extended control Google AI Overviews?
No. Google-Extended is a robots.txt token that decides whether Google may use your content to train Gemini models and ground Gemini apps, and Google states it does not affect inclusion or ranking in Google Search. AI Overviews are built from ordinary Googlebot crawling; to leave them, use Search Console’s Search generative AI setting, which has applied to every site since 31 August 2026.
What are user-triggered fetchers, and why can robots.txt not block them?
A user-triggered fetcher retrieves a page because a person asked an assistant about it, and most vendors treat that as browsing rather than crawling. OpenAI says robots.txt rules "may not apply" to ChatGPT-User and Perplexity says Perplexity-User "generally ignores" them, while Anthropic’s Claude-User and Mistral’s MistralAI-User respect robots.txt. These rows are marked "Robots can't block", because only a CDN or firewall can turn such a bot away.
How accurate is the live probe?
The probe is evidence, not proof, and every result says so. It sends each bot’s documented user agent from RankX AI’s servers, so a user-agent rule shows up exactly, but a network that verifies crawlers by IP address or Web Bot Auth may refuse the probe and admit the real crawler. That is why each result carries two control fetches, a browser and a plain curl, and reads "as seen from our probe".
My result says generic bot protection. What does that mean?
Your site accepted the browser control but refused the plain curl control and every bot user agent alike, so no per-bot verdict would be honest. The fix is in your CDN’s bot settings: decide per purpose which AI crawlers you want, and allow the search-purpose ones explicitly if AI visibility matters to you.
Should I block AI training crawlers?
That is a policy choice, and the tool leaves it to you. Blocking training bots such as GPTBot, ClaudeBot, meta-externalagent and CCBot keeps your content out of future model training without touching AI search, and Cloudflare said in September 2026 that 17% of sites on its network block AI training in some way, while fewer than 1% block search bots. Do it per purpose, because blanket-blocking everything with AI in the name also removes you from AI platforms like ChatGPT search, Perplexity and Claude.
Which AI crawlers can nobody verify?
ByteDance’s Bytespider and xAI’s Grok. ByteDance publishes no documentation for Bytespider, and xAI documents no crawler token at all, so robots.txt cannot even target it. Both show as "compliance unverifiable" rather than an invented verdict.
Can I block AI training but keep AI search visibility?
Yes. Disallow GPTBot, ClaudeBot, meta-externalagent, Amazonbot, CCBot, MistralAI-Training, Google-Extended and Applebot-Extended, and add no rules for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or DuckAssistBot; the snippet below the checker is ready to copy. On Cloudflare, choose Disallow AI Training rather than Block, because Block now turns Googlebot away too, then re-run the check.
How is this different from a robots.txt checker or tester?
A generic robots.txt tester checks one user agent against one URL, and Google’s robots.txt report and URL Inspection tool answer for Google alone, not for search engines and AI crawlers together. The AI Crawler Access Checker works out the rule that governs every documented AI bot at once. It then requests your homepage as each of them, because a CDN can turn away a bot your robots.txt allows.
How often should I re-check my site?
After any CDN, WAF or robots.txt change, and quarterly otherwise, because vendors add bots and CDNs change their settings. Cloudflare changed its own on 15 September 2026, when choosing Block for AI training began to stop Googlebot, Applebot and Bingbot as well. RankX AI re-reads your robots.txt, llms.txt and homepage every day on every account and reports when crawler access gets worse.
Does Cloudflare block AI crawlers by default?
Not the AI search crawlers, which decide whether an assistant can cite you. Since 15 September 2026 a new domain is offered one of two presets: with no ads, Search, Training and Agent are all allowed; with ads, Search stays allowed, Training is set to Disallow AI Training and Agent is blocked on pages with ads. What a live check often finds instead is a generic bot rule that refuses every non-browser request, the search crawlers included.
Will blocking AI training crawlers also block Googlebot?
At Cloudflare, only if you choose Block: since 15 September 2026, Block and Block on pages with ads for AI training "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot". Disallow AI Training refuses training through robots.txt and keeps those three crawling for search, and existing training blocks were migrated to it. In robots.txt the rules stay independent, so disallowing GPTBot has never affected Googlebot.
How do I check which AI bots can access my website?
The quickest way to check is to enter your domain above: the tool reads your robots.txt under RFC 9309 and then requests your homepage once as each documented AI bot, in about twenty seconds. Doing it by hand means working out which robots.txt group applies to each token, then repeating a curl with each user agent against a browser control. The tool needs no account and keeps no result beyond a 30-minute cache.
Put the evidence to work
Use an AI crawler access result to choose the next check.
Read the robots.txt rule and the observed response together, for the AI crawlers and for Googlebot and Bingbot. An allowed rule cannot establish that a bot reached every page; a blocked response needs investigation at the affected URL.
Search access and training are different choices.
OpenAI documents OAI-SearchBot and GPTBot as separate controls for search and model training. Check the named user-agent and purpose before changing a rule. Do not apply one blanket “allow AI” decision to every crawler.
Recheck the page after a fix.
If your CDN or firewall challenges the probe, investigate that response with your hosting team rather than assuming robots.txt is the only block. Recheck after a reviewed change. Access is the first step; whether assistants then name and cite you is what the AI Visibility Checker shows.
Guidance reviewed
Start here
See where you show up in AI answers today.
Start with your website. RankX AI recognises your platform, suggests the questions to track and audits your pages, with first results minutes after setup.