The AI Crawler Access Checker shows which AI bots you block.

The AI Crawler Access Checker is a free robots.txt checker for AI crawlers. For every documented AI bot, from GPTBot to ClaudeBot, it reports what your robots.txt asks, parsed under RFC 9309, and what your server actually returns to that bot’s user agent, keeping policy and evidence apart because on many sites they disagree.

Because robots.txt is a request, not a lock. Your robots.txt can allow GPTBot while your CDN, WAF or bot rules turn the same bot away at the edge with a 403, a challenge or a 402 payment demand, and nothing on your side errors or logs a complaint. That gap, robots allows but the edge blocks, is the single most common surprise this checker finds, and only a live request can reveal it.

No. GPTBot is OpenAI’s training crawler; ChatGPT search results come from a different bot, OAI-SearchBot, and pages a ChatGPT user asks about are fetched by a third, ChatGPT-User. Blocking GPTBot stops your content feeding future model training while leaving you visible in ChatGPT search. Blocking OAI-SearchBot is what removes you from ChatGPT search results. The checker reports each of the three separately.

No, and this is the most common misreading in AI crawler control. Google-Extended is a robots.txt token, not a crawler, so it never appears in your logs. Google-Extended decides whether Google may use your content to train Gemini models, including the models behind AI Overviews and AI Mode, and to ground Gemini apps. Google states it does not affect inclusion or ranking in Google Search. AI Overviews are built from ordinary Google Search crawling by Googlebot, and there is no separate AI Overviews bot. Search Console's Search generative AI setting, under Settings, removes a site from AI Overviews and AI Mode and leaves its ordinary results untouched. It has applied to every site since 31 August 2026. For single pages, nosnippet, max-snippet and noindex still limit what an AI Overview may use, and each one also changes the ordinary result.

A user-triggered fetcher retrieves a page because a human asked an assistant about it, and most vendors treat that as the user browsing rather than a crawl. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores" them, and Meta and Amazon say much the same for theirs. The honourable exceptions are Anthropic’s Claude-User and Mistral’s MistralAI-User, which respect robots.txt even for user requests. The checker marks these rows "Robots can't block" so a robots.txt rule is never mistaken for a control; only a CDN or firewall can actually turn such a bot away.

The probe is evidence, not proof, and the checker says so on every result. It sends each bot’s documented user agent from RankX AI’s own servers, so an edge rule keyed on user agent strings shows up exactly. But a growing number of networks verify crawlers by IP address or Web Bot Auth, and those will refuse our probe while admitting the real crawler, which can read as blocked when the bot is fine. That is why every result also carries two control fetches, a normal browser and a plain curl, and why the verdict is always "as seen from our probe", never a guarantee.

Generic bot protection means your site accepted the browser control but refused the plain curl control and every bot user agent alike. When everything non-browser is blocked, no per-bot verdict is honest: the checker cannot tell whether GPTBot specifically is blocked or whether all automation is. The fix is to check your CDN’s bot settings, decide per purpose which AI crawlers you want, and allow the search-purpose ones explicitly if AI visibility matters to you.

That is a policy choice, not a technical one, and the checker deliberately does not make it for you. Blocking training bots such as GPTBot, ClaudeBot, meta-externalagent and CCBot keeps your content out of future model training without touching your visibility in AI search, because search runs on different bots. Plenty of sites make that call: Cloudflare said in September 2026 that 17% of the sites on its network block AI training in some way, while fewer than 1% block search bots. What the checker insists on is doing it per purpose: blanket-blocking everything with AI in the name also removes you from ChatGPT search, Perplexity and Claude’s citations, which is usually not what a marketing site wants.

ByteDance’s Bytespider and xAI’s Grok. ByteDance publishes no documentation for Bytespider at all, and third-party logs repeatedly report it ignoring robots.txt. xAI is worse: it documents no crawler token whatsoever, so robots.txt cannot even target it, and the names circulating online are unofficial and contradict each other. The checker shows both as "compliance unverifiable" rather than inventing a verdict, because a made-up green tick would be worth less than an honest unknown.

Yes. Add a robots.txt group per training bot and leave the search bots alone. Disallow GPTBot, ClaudeBot, meta-externalagent, Amazonbot, CCBot, MistralAI-Training, Google-Extended and Applebot-Extended, and do not add rules for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or DuckAssistBot. The page below the checker carries the exact snippet to copy. On Cloudflare, choose Disallow AI Training rather than Block, because Block now turns Googlebot away too. Then re-run the check, because a CDN rule can still override what robots.txt asks.

A generic robots.txt tester reads the file and checks one user agent against one URL. Google’s robots.txt report in Search Console shows which robots.txt files Google found for your top 20 hosts and any parsing errors, and Google points to its URL Inspection tool to test one URL, which answers for Google alone. The AI Crawler Access Checker is a robots.txt checker for AI crawlers: it works out the rule that governs every documented AI bot at once, then requests your homepage as each of them, because a CDN can turn away a bot your robots.txt allows.

Re-check after any CDN, WAF or robots.txt change, and quarterly otherwise, because the ground moves on both sides: vendors add bots and rename tokens through the year, and CDNs change their settings under you. Cloudflare changed its own on 15 September 2026, when choosing Block for AI training began to stop Googlebot, Applebot and Bingbot as well. A site that was open in the spring can be closed by the autumn without anyone touching robots.txt. Leave an email under your result and the checker re-runs once a month, writing only when a crawler gains or loses access.

Not the AI search crawlers, which are the ones that decide whether an assistant can cite you. Since 15 September 2026 Cloudflare offers a new domain one of two presets, chosen by whether the site earns money from ads. With no ads, Search, Training and Agent are all allowed. With ads, Search stays allowed, Training is set to Disallow AI Training, which blocks training-only crawlers such as GPTBot and ClaudeBot, and Agent is blocked on pages that display ads. What is not a default, and what this checker finds most often, is a generic bot rule that refuses every non-browser request, the search crawlers included.

At Cloudflare, only if you choose Block. Since 15 September 2026, Cloudflare says Block and Block on pages with ads for AI training "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot", so either one turns Google away. The same day Cloudflare added Disallow AI Training, which refuses training through robots.txt and keeps those three crawling for search, and it migrated existing training blocks, the 2025 one-click toggle included, to that setting. In robots.txt the rules stay independent: disallowing GPTBot has never affected Googlebot, and Google-Extended is a separate token again. This is a CDN behaviour, which is why a robots.txt tester cannot see it and a live fetch can.

Enter your domain above to check if AI crawlers can access your website. The AI Crawler Access Checker does both halves in about twenty seconds: it reads your robots.txt the way a crawler reads it, under RFC 9309, and it then requests your homepage once as each documented AI bot to see what your server actually returns. Doing it by hand means fetching your robots.txt, working out which group applies to each documented token, and then repeating a curl with each user agent while comparing against a browser control to tell a per-bot block from blanket bot protection. The tool needs no account and stores no result beyond a 30-minute cache.