Skip to content
RankX AI

Cookie settings

Choose which cookies we can use. You can change this at any time from “Cookie settings” in the footer.

Essential

Security, bot protection and remembering your choice. The site needs them to run, and they do not identify you. Always on.

Google Analytics cookies that tell us which pages are read and which links are followed, so we can improve them. Off until you switch it on.

Tells advertisers when their advert led to your visit. Includes the Meta (Facebook) pixel and the LinkedIn Insight Tag, which set cookies on this domain (LinkedIn also on linkedin.com) and can be told if you later start a trial.

More in the cookie policy and the privacy policy.

AI Search Essentials

Block AI Crawlers at Cloudflare, Lose Googlebot

Since 15 September 2026, Block on Cloudflare's AI training control also stops Googlebot, Applebot and Bingbot. The setting that keeps search, and what migrated.

Published Last updated 8 minute read
Decision diagram: Crawler rule, Training bot, Search bot, Check access.
On this page 10 sections
  1. Does Cloudflare block AI crawlers?
  2. What changed at Cloudflare on 15 September 2026?
  3. Why does Block on AI training now block Googlebot?
  4. Did Googlebot lose crawl access before 15 September?
  5. How do you block AI crawlers using Cloudflare without losing search?
  6. Which vendor controls protect search visibility instead of Block?
  7. Do the new-domain presets apply to your existing site?
  8. Are Verified bots still allowed by default?
  9. How do you check what your edge is actually doing?
  10. Verify product access at the edge before changing bot policy

Short answer: Cloudflare blocks AI crawlers only when you choose to, under Security Settings, Configure AI bot policies. Since 15 September 2026, choosing Block for AI training also stops Googlebot, Applebot and Bingbot, because each crawls for search and training at once. Disallow AI Training refuses training and keeps search, and existing training blocks were migrated to it.

Does Cloudflare block AI crawlers?

Cloudflare blocks AI crawlers only when a site owner sets it to, and the setting lives under Security Settings, then Configure AI bot policies. Cloudflare sorts AI traffic into three behaviours, Search, Training and Agent, and each gets its own policy.

Category

Cloudflare's definition

Who it usually covers

Search

Any behaviour that collects or indexes your content, so it can answer questions about it later

Googlebot, Bingbot, OAI-SearchBot

Training

A crawler taking your content to train or fine-tune a model

GPTBot, ClaudeBot, meta-externalagent

Agent

Automated activity acting in real time on a person's behalf

ChatGPT-User, Perplexity-User, browser-use agents

An existing zone keeps its own configuration. Cloudflare's bot management still blocks non-verified bots by default, and the AI policies decide what happens to verified ones, so website owners choose which AI companies may scrape their content.

What changed at Cloudflare on 15 September 2026?

Three things changed. Block now reaches crawlers that do more than one job, a new setting called Disallow AI Training refuses model use without refusing the crawler, and new domains are offered one of two presets.

That control now has four settings, and the difference between the last two is the one that costs sites their Google traffic.

Training setting

What it does

Effect on Googlebot, Applebot and Bingbot

Allow

Nothing is refused, unless a WAF rule says otherwise

Crawl for search and model use

Disallow AI Training

Publishes a no-training rule in robots.txt and blocks training-only crawlers at the edge

Keep crawling for search

Block on pages with ads

Refuses those crawlers on pages detected to serve ads

Refused on those pages, search included

Block

Refuses every crawler in the category

Refused everywhere, search included

Disallow AI Training is the setting most sites want. Cloudflare says it is "named for the Disallow: directive it publishes in your robots.txt", and it still blocks the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI at the edge, because blocking those does not affect search.

The new-domain change is narrow. From 15 September 2026, a domain onboarding to Cloudflare is offered a preset chosen by whether the site earns money from advertising. With no ads, Search, Training and Agent are all Allow; with ads, Search stays Allow, Training is Disallow AI Training and Agent is Block on pages with ads.

What happened to the legacy Block AI bots toggle?

Block AI bots was deprecated on 15 September 2026, and Cloudflare migrated existing settings rather than leaving the old toggle to enforce. Its summary for existing customers is "Nothing, in almost every case."

A legacy block, full or ads-only, became Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent. A zone that had set the granular controls kept its Search and Agent choices, and Block became Disallow AI Training. Either way the search crawlers keep crawling.

So the sites exposed are the ones that choose Block from 15 September on, believing it is the stronger version of the same thing.

Why does Block on AI training now block Googlebot?

Because Googlebot is a mixed-use crawler: one bot that indexes for search and also collects data for AI models. Cloudflare's post says Block and Block on pages with ads "now apply to all training crawlers, including mixed-use crawlers", and names Applebot, Bingbot and Googlebot.

Choose the next step
Inspect the category rule and the actual crawler
  1. Rule category

    A broad category can include more than one use.

  2. Crawler identity

    Confirm which crawler the request belongs to.

  3. Edge result

    Check the real allow or block response.

A setting label is not a substitute for verifying search-crawler access.

Under Disallow AI Training, Cloudflare says, they "can keep crawling your site for search. Selecting Block stops them entirely."

Cloudflare's bot report of 1 July 2026 puts mixed-use crawlers at over 36 percent of crawler activity and says more than 20 percent of the web sits behind its network. W3Techs puts Cloudflare on 25.7 percent of all websites and 85.0 percent of sites whose reverse proxy is known, as of September 2026.

The same report puts Google at roughly 88 percent of referral traffic, the source of almost all the human visitors still arriving from search.

Did Googlebot lose crawl access before 15 September?

Yes. Three site owners reported Googlebot taking HTTP 403 responses from this setting during August 2026. All three accounts are self-reported, not measured, but they describe the same failure.

In early August a site owner reported that, with this setting on Block, Googlebot and Bingbot got 403 responses on the sitemap; Search Engine Journal covered it on 4 August 2026 and Playwire on 5 August. Bot Fight Mode was also on, so neither setting was isolated.

A Cloudflare Community thread from 25 August 2026 is cleaner: 403s on 68 URLs between 15 and 22 August, 47 of them canonical sitemap URLs, after ai_training=block was set with Search and Agent blocking off. Indexed pages fell from 114 to 72, and access returned when the setting went back to Allow on 24 August, while a second thread from 22 August reports verified Googlebot 403s from early July.

Choose Disallow AI Training and leave Search on Allow. That refuses model use, stops the training-only bots at the edge and keeps every search crawler able to index your pages.

  1. Open the policies

    In the Cloudflare dashboard, open your domain, then Security Settings, then Configure AI bot policies.

  2. Leave Search on Allow

    Search covers Googlebot, Bingbot and the AI search crawlers that decide whether assistants can cite you.

  3. Set Training to Disallow AI Training

    This writes a no-training rule into robots.txt through Bot Preference Sync and blocks training-only crawlers.

  4. Choose an Agent setting

    Allow keeps assistants able to fetch a page a user asked about. Block on pages with ads protects ad-funded pages.

  5. Block named bots only where needed

    Use AI Crawl Control to allow or block an individual crawler, or a WAF custom rule for scrapers that ignore robots.txt.

What does AI Crawl Control do?

AI Crawl Control is a separate surface from the three policies. Cloudflare describes it as letting you "see which AI services access your content" and "set allow or block rules for individual crawlers", so it is where you review AI crawler activity and act on one named bot.

When Bot Preference Sync writes your robots.txt, it adds the Content Signals Policy, with signals such as search=yes, ai-train=no and a use value of immediate, reference or full. Those signals state what a crawl may be used for, which allow and disallow rules cannot express. The robots.txt rules it writes name training crawlers such as GPTBot, ClaudeBot, Amazonbot, Bytespider and Google-Extended, and Bot Preference Sync replaced the older Managed robots.txt feature on 15 September 2026.

From RankX AICheck the public-page access evidence.Keep checkout, account and private data protections separate from public catalogue access.Use the AI crawler access checker

Which vendor controls protect search visibility instead of Block?

Say what you mean in robots.txt, where the search engines let you refuse AI training without giving up their search results. Every crawler Cloudflare names as mixed-use publishes a purpose-specific control of its own.

Crawler

The vendor's own AI control

Where it goes

What the vendor says about search

Googlebot

robots.txt

"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Applebot

Applebot-Extended

robots.txt

"Webpages that disallow Applebot-Extended can still be included in search results."

Bingbot

NOARCHIVE, with NOCACHE as a partial

A robots meta tag, per page

"Content with the NOCACHE tag or NOARCHIVE tag will still appear in our search results."

Do not treat the Bing row as equal to the other two. NOCACHE is partial: Microsoft says "only URLs, Titles and Snippets may be used in training". NOARCHIVE is the full refusal, and content tagged with it "will not be included in Bing Chat answers, not be linked to in the answers".

Bing is also the gap in Cloudflare's version. Microsoft is building support for a robots.txt no-training preference, targeted for early 2027, and until then Cloudflare says Disallow AI Training "will not automatically convey a no-training preference to Bing".

That is the whole asymmetry. Stated in robots.txt, refusing AI training is free. Stated as Block at the edge, it costs you Googlebot, Applebot and Bingbot.

When is blocking at Cloudflare the right answer?

Block at the edge when the crawler offers no purpose-specific control of its own. That covers training crawlers that give you no option, malicious bots and any scraper that never identifies itself honestly, and bots that read robots.txt and then ignore it. There, an edge rule is how you protect content and site performance from crawling without consent.

There is also a third position, in closed beta. Pay per crawl lets a site owner set one price for the zone and charge AI crawlers per request, answering anything unpaid with HTTP 402. That is a decision for content creators about monetising original content, not about AI visibility.

Do the new-domain presets apply to your existing site?

No. Cloudflare scopes the presets to new domains onboarding to Cloudflare, which is narrower than much of the coverage says. Its 15 September post says the new controls themselves "are available to all customers, on all plans".

TechCrunch, reporting the July 2026 announcement, wrote that the default changes "will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says". That is the version that spread, and Cloudflare's own pages do not state it.

We cannot resolve the difference from outside. Open Security Settings, find Configure AI bot policies, and read what your zone says.

Hostname, not page

The 2026 post and the dashboard describe blocking on pages that display ads, while the post that shipped the feature on 1 July 2025 said it would "block AI bots ONLY on that hostname". Until Cloudflare reconciles the two, assume one ad on a hostname can make Block on pages with ads act site-wide.

Are Verified bots still allowed by default?

No. Cloudflare's wording: "non-verified bots are still default blocked, but we are no longer viewing Verified as 'default allowed.'" A Verified Search crawler reaches your site because you allow Search, not because it is verified.

Cloudflare's verified bot documentation lists eleven behavioural categories, and only Search, Training and Agent carry these policy controls. A crawler whose only behaviour is SEO, such as a site auditor, is caught by none of them.

How do you check what your edge is actually doing?

Check the edge and the file separately, because they are two systems and neither reports on the other. Five checks, in the order that finds problems fastest.

How it works
Test what the edge actually serves
  1. Select URL

    Use an important public page.

  2. Inspect request

    Check crawler identity and response evidence.

  3. Compare intent

    Confirm the result matches the policy you wanted.

Use observed access results before assuming a dashboard setting is working.

  1. Open Security Settings, then Configure AI bot policies, and read Search, Agent and Training. If Training says Block, you are blocking Googlebot, Applebot and Bingbot.
  2. If you used the legacy Block AI bots toggle, confirm it migrated to Disallow AI Training for Training.
  3. Look in Search Console and Bing Webmaster Tools for sitemap fetch failures. A 403 on a sitemap is the symptom that surfaced this in August.
  4. Run the free AI Crawler Access Checker. It parses your robots.txt under RFC 9309, then requests your homepage as each of 27 crawlers, with a browser and a plain curl as controls, so a per-bot block can be told apart from blanket bot protection.
  5. Compare the two answers. Where robots.txt allows a crawler and the live fetch refuses it, the edge is overriding your file.

A crawler refused at the perimeter never reaches your server, so nothing in your analytics records the visit that did not happen. When an assistant never cites you, the cause is sometimes a 403 nobody logged, and AI bot tracking shows how to read those requests in your logs.

On WordPress, check which robots.txt is served before you edit anything, because a physical file on disk silently replaces every rule your SEO plugin generates. Blocking AI crawlers in WordPress covers it.

Access is the floor, not the goal. A crawler that gets a clean 200 still has to find your words in the raw HTML, and how AI crawlers read your site has the per-bot detail.

Verify product access at the edge before changing bot policy

Choose a public category page, a public product and its main image. Record the Cloudflare setting and the response each receives, including a verified search crawler request where you have the tools.

Leave login, cart and checkout protection untouched while you test. If a policy blocks a crawler you meant to allow, make the narrowest correction, keep a note of the previous setting and retest the same URLs.

A response that is now allowed confirms access, not that the product was indexed or chosen in an AI answer. Watch the search evidence after the fix.

Next, understand crawler purposes or verify crawler identity.

Questions about AI Search Essentials

Does Cloudflare block AI crawlers by default?

Not on an existing site: an existing zone keeps its own configuration. From 15 September 2026, a new domain onboarding to Cloudflare is offered one of two presets, and Search is allowed in both. An ad-supported site's preset sets Training to Disallow AI Training and Agent to Block on pages with ads.

Does blocking AI training in Cloudflare block Googlebot?

Only if you choose Block or Block on pages with ads for Training. Since 15 September 2026 those stop Googlebot, Applebot and Bingbot as well, search included, while Disallow AI Training refuses training through robots.txt and leaves them crawling for search. In robots.txt the rules stay independent: disallowing GPTBot has never affected Googlebot.

Does the Cloudflare AI crawler change apply to free plans?

Cloudflare says the new AI traffic controls are available to all customers on all plans, including the Free tier. The new-domain presets are scoped to new domains onboarding to Cloudflare, though some write-ups report they reach existing Free zones too. Read your own zone rather than any summary.

Does my robots.txt override Cloudflare bot blocking?

No. An edge rule refuses the request before your server sees it, and robots.txt is a file on that server stating a preference. A crawler blocked at Cloudflare cannot read the permission you wrote for it.

Sources

  1. Cloudflare Blog, Have it both ways (15 September 2026): Disallow AI Training, mixed-use crawlers, presets and migration (opens in a new tab) Checked 2026-10-26.
  2. Cloudflare Blog, Your site, your rules: new AI traffic options (1 July 2026) (opens in a new tab) Checked 2026-10-26.
  3. Cloudflare docs, Block AI Bots and Configure AI bot policies (opens in a new tab) Checked 2026-10-26.
  4. Cloudflare docs, Verified bots and the eleven behaviour categories (opens in a new tab) Checked 2026-10-26.
  5. Cloudflare Blog, Control content use for AI training (1 July 2025): hostname-level ad detection (opens in a new tab) Checked 2026-10-26.
  6. Cloudflare, Content Independence Day one year on (1 July 2026): mixed-use crawler share and referral traffic (opens in a new tab) Checked 2026-10-26.
  7. Cloudflare changelog, New options to manage AI traffic (1 July 2026) (opens in a new tab) Checked 2026-10-26.
  8. Google Search Central, Google's common crawlers: Google-Extended and Search (opens in a new tab) Checked 2026-10-26.
  9. Apple, About Applebot and Applebot-Extended (opens in a new tab) Checked 2026-10-26.
  10. Bing Webmaster Blog, NOCACHE and NOARCHIVE for Bing Chat (22 September 2023) (opens in a new tab) Checked 2026-10-26.
  11. OpenAI, Overview of OpenAI crawlers (opens in a new tab) Checked 2026-10-26.
  12. Search Engine Journal (4 August 2026): community report of Googlebot 403s, not a study (opens in a new tab) Checked 2026-10-26.
  13. W3Techs, Usage statistics of Cloudflare (September 2026) (opens in a new tab) Checked 2026-10-26.
  14. Cloudflare Community (25 August 2026): 403s on 68 URLs, self-reported (opens in a new tab) Checked 2026-10-26.
  15. Cloudflare Community (22 August 2026): verified Googlebot 403s, self-reported (opens in a new tab) Checked 2026-10-26.
  16. Vercel and MERJ, The rise of the AI crawler (17 December 2024) (opens in a new tab) Checked 2026-10-26.
  17. Cloudflare docs, robots.txt setting and Bot Preference Sync, with Content Signals (opens in a new tab) Checked 2026-10-26.
  18. Cloudflare docs, Bot custom rules (opens in a new tab) Checked 2026-10-26.
  19. Playwire (5 August 2026): second-day coverage of the same community report (opens in a new tab) Checked 2026-10-26.
  20. Cloudflare docs, What is Pay per crawl (closed beta, HTTP 402) (opens in a new tab) Checked 2026-10-26.
  21. Cloudflare docs, AI Crawl Control (opens in a new tab) Checked 2026-10-26.
  22. Cloudflare docs, Manage AI crawlers: per-crawler actions and robots.txt violations (opens in a new tab) Checked 2026-10-26.
  23. TechCrunch (1 July 2026): reports a free-tier scope Cloudflare's own pages do not state (opens in a new tab) Checked 2026-10-26.

Written by

Asif Syed Founder & CEO

Asif Syed is the founder and CEO of RankX AI, the AI search visibility platform. He builds the product and writes here about GEO, AI search measurement and WordPress.

CoversThis article covers the AI Search Essentials topic, the Website Audit feature and the AI Crawler Access Checker tool.

Terms usedAI Crawler, GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, Robots.txt and PerplexityBot.

Read this page asMarkdown: /blog/cloudflare-block-ai-crawlers.md.

All articlesEverything RankX AI publishes is listed on the blog index.

Ask an assistantAsk ChatGPT (opens in a new tab), Ask Claude (opens in a new tab) or Ask Perplexity (opens in a new tab).

Preferred sourceIf Google is your front door, you can add RankX AI as a preferred source (opens in a new tab), which asks your own results to surface more of what we publish.

Back to the top

Start here

Check the public-page access evidence.

Keep checkout, account and private data protections separate from public catalogue access.

7-day free trial. No credit card required. On Shopify, install the app and Shopify bills your plan.