How to Block AI Crawlers in WordPress

How do you block AI crawlers in WordPress?

Blocking AI crawlers in WordPress means naming each crawler's robots.txt token and disallowing it, or refusing the request outright at your CDN. WordPress builds robots.txt on the fly rather than storing it as a file, so the four practical routes are an SEO plugin's robots.txt editor, a dedicated blocking plugin, a physical file you write yourself, or Cloudflare sitting in front of the site.

The default WordPress robots.txt is three lines. WordPress generates User-agent: *, a Disallow for the admin directory and an Allow for admin-ajax.php, and nothing in it mentions AI crawlers at all. Every AI rule on a WordPress site was added by somebody or by something.

The four routes, and when each is right

• An SEO plugin's editor. Yoast, Rank Math and AIOSEO all expose a robots.txt screen. Best when you want a handful of rules you chose, and the change lives with the plugin rather than in a file you might lose in a migration.
• A dedicated blocking plugin. Block AI Crawlers and the Raptive Ads plugin ship maintained lists. Best when you want everything blocked and do not want to track a moving roster of tokens yourself. Read the audit section below before installing one.
• A physical robots.txt in the site root. Best when your rules must survive a plugin being deactivated. It has a consequence most guides do not mention, and it is the subject of the next section.
• Cloudflare in front of the site. The only one of the four that ENFORCES rather than requests, so it is the only one that reaches a crawler which ignores robots.txt. Free plans included. It has its own trap, and it gets a full section below.

Whichever route you take, a WordPress AI crawler rule is only worth what the crawler chooses to do with it. Robots.txt is a published request, and the vendors differ on whether they honour it for user-triggered fetches. That is covered further down.

Crawler access is the first layer of a larger question. The rest of it sits in WordPress SEO in the AI era: what the SEO plugins actually emit, whether page builders hide content, and what we found 359 live WordPress sites serving.

Should a WordPress site block AI crawlers at all?

Block the training crawlers if you want to, and leave the search crawlers alone. That is the position this article argues for, and it is the one most WordPress owners already hold once the question is put to them precisely: little objection to being cited, a real objection to becoming training data.

The case for blocking training crawlers is sound and worth stating plainly, because most of the coverage treats it as either obvious or indefensible. A training crawler takes your posts, sends no visitor back, and feeds a model that may answer your reader's question without naming you. On a large WordPress site it costs real bandwidth too. A publisher who disallows every one of them has made a defensible choice.

The case against blocking the search crawlers is the one the pasted lists obscure. A search crawler is how an assistant learns you exist well enough to cite you. Disallowing it removes the citation and leaves the page exactly as copyable as it was before, because a public page stays public either way. That is the trade the plugins make on your behalf without describing it.

Our own position, since it is fair to declare it: rankxai.com allows every crawler in the roster, training bots included, and names each one explicitly rather than leaning on the wildcard. We sell AI visibility, so disallowing GPTBot while publishing about generative engine optimisation would be indefensible, and our own readiness checker would flag our own site.

A publisher whose income comes from readers rather than from software should weigh the same evidence differently, and reach a different answer without being wrong. The general version of that argument sits in what AI crawlers actually read on your site.

Why WordPress robots.txt edits stop working without warning

A physical robots.txt file in the WordPress root silently disables every plugin rule you have written. WordPress generates robots.txt only when the request reaches WordPress, and a request only reaches WordPress when no file of that name exists on disk.

The mechanism is in WordPress's own default rewrite rules. They carry RewriteCond %{REQUEST_FILENAME} !-f, which means the rewrite to index.php runs only if the requested filename is not a real file. Create robots.txt and the web server answers the request itself.

What breaks then is the whole chain. WordPress never runs do_robots(), so it never applies the robots_txt filter, and that filter is precisely what Yoast, Rank Math, AIOSEO and the AI-blocking plugins hook into. The plugin screen still shows your rules, because it shows what it would generate rather than what your server serves.

The Block AI Crawlers plugin treats this as serious enough to refuse installation. Read on 9 September 2026, its source refuses activation when a physical file is present and tells the owner to rename that file before trying again. A plugin author blocking their own install over a mechanism is a fair signal that the mechanism catches people.

Caching is the second WordPress trap here. Managed hosts cache robots.txt aggressively, and WordPress VIP documents caching it for long periods with a dedicated purge step, so a correct edit can take hours to appear. Verify by fetching the URL rather than by reading the plugin screen, and purge before concluding that an edit failed.

What the standard WordPress AI block lists actually disallow

The block lists WordPress plugins install disallow the crawlers that decide AI citations, not only the crawlers that collect training data. Every page on this search result hands the reader a list to paste, and none of them audits the list it is handing over, so we did.

On 9 September 2026 we took the two lists WordPress owners actually install, both fetched from source rather than quoted. The ai.robots.txt community list (opens in a new tab) carries 175 user-agent tokens, and the plugins advertising 75 or more blocked bots are generated from it. The list inside the Block AI Crawlers plugin carries 163.

Anyone can re-run this. Fetch the community list from its repository and the plugin's generated list from the plugin's, pull out every User-agent line, and check each token against the vendor documentation linked beside every bot in our AI crawler access checker. It needs no account and no paid tool, which is the point: the finding is checkable rather than asserted.

The community list against a roster of 25 documented crawlers

So the WordPress plugin marketed as blocking AI scrapers removes the site from AI answers as well. That may be exactly what a publisher wants, and the ai.robots.txt project is honest about being a total opt-out. The gap is in the WordPress framing around it, where the same list is presented as protecting content while it is also cancelling citations.

Which tokens on the WordPress block lists do nothing

Fourteen tokens on the community list cannot do what a WordPress owner pasting them expects, for two separate reasons. Four name crawlers that no vendor documents any more, and ten are not valid robots.txt product tokens because they contain a space.

Four tokens on the lists that no vendor documents

anthropic-ai and Claude-Web are both on the lists, and Anthropic's crawler documentation names ClaudeBot, Claude-User and Claude-SearchBot. FacebookBot is on the lists, and Meta documents FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. None of those three matches a crawler either vendor currently documents, so the rules are inert.

Cohere is the sharpest case, because the list gets it backwards. Coherebot is the only token Cohere's own crawler documentation tells you to disallow, and it is absent from the list, while two tokens Cohere does not document, cohere-ai and cohere-training-data-crawler, are both on it.

Twenty-one tokens the standard does not allow

RFC 9309 (opens in a new tab), the robots.txt standard, states that a product token must contain only uppercase and lowercase letters, underscores and hyphens. Twenty-one of the 175 tokens fall outside that, and ten of them contain a space: Brightbot 1.0, ChatGPT Agent, Datenbank Crawler, Echobot Bot, Kangaroo Bot, Linguee Bot, netEstate Imprint Crawler, Poseidon Research Crawler, Sidetrade indexer bot and Terra Cotta.

A token with a space in it has no defined behaviour. What a given crawler does with User-agent: Kangaroo Bot is left to that vendor's parser rather than settled by the standard, and no vendor documents the answer. Three further pairs are the same token in two cases, which the standard's case-insensitive matching makes redundant rather than harmful.

Six of the twenty-one are outside the grammar only because they contain a digit, and several of those are the names their vendors publish. That is the standard being narrower than practice rather than the list being careless, and it is worth knowing before you assume a rule is working: the tokens with a space in them are the ones with nothing behind them at all.

Training crawlers and search crawlers answer different questions

A training crawler decides whether your content trains a future model, and a search crawler decides whether your page can be cited in an answer today. Most WordPress robots.txt advice treats the two as one category called AI bots, and that is what makes the pasted lists so much broader than their readers expect.

OpenAI documents the split plainly. GPTBot crawls content for training generative AI models. OAI-SearchBot surfaces websites in ChatGPT search results, and OpenAI's bot documentation (opens in a new tab) states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Anthropic, Perplexity, Meta, Mistral and Amazon all draw the same three-way split between a training crawler, a search crawler and a user-triggered fetcher. The tokens differ and the compliance statements differ, which is the whole reason a pasted list is a poor substitute for a decision. The roster of every documented crawler sits in what AI crawlers actually read on your site.

The asymmetry worth knowing on WordPress: blocking a training crawler is reversible in effect but not retroactive, since content already inside a released model cannot be withdrawn. Blocking a search crawler takes effect on the next crawl and reverses on the one after that. One decision is close to permanent and the other is a toggle.

Blocking Google-Extended does not remove WordPress pages from AI Overviews

Google-Extended controls Gemini, not AI Overviews, and almost every WordPress robots.txt guide gets this wrong. Google documents Google-Extended as a control over whether crawled content trains and grounds Gemini models, and states that it does not impact a site's inclusion in Google Search and is not a ranking signal.

AI Overviews and AI Mode are part of Google Search. Google's own documentation on AI features (opens in a new tab) states that robots.txt directives for Googlebot are the control for how sites are crawled for Search, and that limiting what appears is done with nosnippet, data-nosnippet, max-snippet or noindex.

Google-Extended also never appears in a WordPress access log, because it is a control token rather than a crawler and has no user agent string of its own. Blocking Gemini grounding is a real choice with real consequences, covered in Google Gemini SEO. It is simply not the lever for Google AI Overviews.

Which AI crawlers ignore a WordPress robots.txt

The user-triggered fetchers are the ones that state they may ignore robots.txt, and the vendors say so in their own documentation rather than leaving it to be discovered. A WordPress robots.txt is a published request, and treating it as enforcement is the mistake that leads to a surprised look at a log file.

Anything a WordPress owner needs enforced belongs a layer up, at the firewall or the CDN. Cloudflare, a web application firewall (WAF) or a host's bot rules can refuse a request outright, which robots.txt cannot. The trade is that enforcement blocks by signature and can catch a crawler you wanted, so it needs the same token-by-token reading as the robots.txt route. The next section covers the Cloudflare route in full, including the setting that quietly takes Googlebot with it.

Your WordPress host may already be doing some of this. Search Engine Land reported on 6 May 2026, from log data collected between 4 and 10 April 2026, that WP Engine enforced platform-level rate limiting on certain bots that customers could not selectively disable. Host behaviour changes, so treat that as an observation with a date on it and confirm your own host's current position before concluding anything about your site.

How to block AI crawlers with Cloudflare

Cloudflare refuses the request at the edge instead of asking the crawler to behave, which is the one thing a WordPress robots.txt cannot do. That makes it the only route in this article that reaches the three fetchers in the table above whose vendors document that robots.txt may not apply, and Bytespider, whose compliance nobody can verify.

AI Crawl Control, which Cloudflare renamed from AI Audit, is available on every plan including the free one. Each crawler takes one of three actions, allow, block or charge, and a block answers 403 Forbidden or, on paid plans, 402 Payment Required carrying a licensing message you write.

The feature worth the detour for a WordPress owner is the robots.txt violations column. It counts the times each crawler fetched something your robots.txt disallowed, which turns whether a bot obeys from a question about vendor documentation into a measurement of your own site. Nothing in WordPress reports that.

Cloudflare will also write the robots.txt for you, and its managed list is better curated than the community one audited above. It disallows eight tokens, every one a training crawler: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. No search crawler appears on it. It prepends a Content-signal line too, search=yes, ai-train=no, the emerging machine-readable form of the same instruction.

The steps, in the Cloudflare dashboard

1. Select the domain. These controls are per zone, so a WordPress multisite spanning several domains needs configuring on each one separately.
2. Open AI Crawl Control. Read the crawler list before changing anything: it shows every AI crawler that has actually reached your site, its category, its request count and its robots.txt violations.
3. Set an action per crawler. Block the training crawlers. Leave the eight search crawlers on Allow, for the reasons the audit above measures.
4. Set the block response under Settings. 403 Forbidden refuses. 402 Payment Required, on paid plans, refuses and tells the operator how to license the content instead.
5. Optionally turn on the managed robots.txt. In Security Settings, filter by Bot traffic and enable the setting that writes a training block into robots.txt. It prepends its rules to whatever WordPress already generates.
6. Re-fetch your own pages afterwards. A Cloudflare rule and a WordPress rule can disagree, and the edge decides.

Read the category setting before you tick it

Cloudflare sorts AI bots into Search, Agent and Training, and its own documentation defines Training as including mixed-purpose crawlers. Because the policy applies the most restrictive rule that matches, selecting Block on Training also blocks Googlebot, Applebot and BingBot. Cloudflare names those three in its announcement.

For a WordPress site that is the most expensive tick on the page, because it removes you from Google Search and from AI Overviews together. It is the same defect as pasting a 175-token list without reading it, moved one layer up the stack and made more costly.

From 15 September 2026 Cloudflare blocks Training and Agent by default on ad-displaying pages for domains onboarding after that date, and leaves Search allowed. Existing customers keep their settings and can confirm no change from Security Settings. The controls live under Configure AI bot policies, where each category takes Block on all pages, Block on pages with ads, or Allow.

What about WooCommerce products and image content?

WooCommerce product pages are read by the same search crawlers that decide AI shopping answers, so a store that pastes a full block list removes its products from the assistants people increasingly buy through. The robots.txt calculus for a shop is the same as for a blog, and the stakes are higher, because the disallowed pages are the ones with revenue attached to them.

The image opt-out directives are a different matter, and the honest answer is that they are not a standard. DeviantArt introduced noai and noimageai on 11 November 2022, describing them as an industry first, and its own announcement claims no adoption by any AI company or standards body. None of the six vendors whose crawler documentation this article cites mentions either directive.

The documented controls apply to images as well as pages. A robots.txt rule stops the crawler fetching the file, and an X-Robots-Tag: noindex response header keeps that file out of search engine indexes. Google-Extended and Applebot-Extended govern training on what those vendors have already crawled. Anything beyond those is a request nobody has agreed to honour.

A WordPress robots.txt that blocks training without losing citations

The rules below disallow the training crawlers and leave the search crawlers alone, which is the configuration most WordPress owners describe when asked what they want. Add them through your SEO plugin's robots.txt editor, or to a physical file if you have one, and read the section above on virtual robots.txt if your edits have not been taking effect.

• User-agent: GPTBot then Disallow: / stops OpenAI collecting your pages for model training.
• User-agent: ClaudeBot then Disallow: / does the same for Anthropic's training crawler.
• User-agent: CCBot then Disallow: / keeps you out of Common Crawl, which many models train from.
• User-agent: Google-Extended then Disallow: / opts out of Gemini training and grounding, and leaves Google Search untouched.
• User-agent: Applebot-Extended then Disallow: / opts out of Apple foundation model training while keeping Applebot's search crawl.
• User-agent: meta-externalagent then Disallow: / opts out of Meta's training crawl.
• User-agent: Amazonbot then Disallow: / opts out of the Amazon crawl that may train Amazon models, and leaves Amzn-SearchBot in place.

Leave the eight search crawlers allowed. They are OAI-SearchBot for ChatGPT, Claude-SearchBot, PerplexityBot, meta-webindexer, Applebot, Amzn-SearchBot, MistralAI-Index and DuckAssistBot. Disallowing any one of them removes the site from that assistant's answers rather than from its training data.

Two rules to resist. Do not disallow Googlebot, which removes the WordPress site from Google Search entirely and from AI Overviews with it. Do not paste a 175-token list without reading it, for the reasons the audit above measures.

A robots.txt says what may be fetched and nothing about what a model should understand. Publishing an llms.txt file alongside it is the complementary move, and the reasoning for both sits under how to rank in AI search.

How to check what your WordPress robots.txt is doing

Fetch the file rather than trusting a plugin screen, because the plugin screen shows what WordPress would generate and the file is what crawlers read. Open your domain with /robots.txt on the end in a browser, or request it from a terminal, and read what actually comes back.

1. Fetch the live file. If it does not match your plugin, a physical robots.txt or a cache is in the way.
2. Read the tokens rather than counting them. Sort them into training, search and user-fetch, and confirm each name against the vendor's current documentation. Four tokens on the popular lists are no longer documented anywhere.
3. Check the eight search crawlers specifically. A Disallow on any one of them is a decision, so make it deliberately rather than by pasting.
4. Confirm Googlebot is allowed in robots.txt AND at your CDN. Cloudflare files it under Training, so the two layers can disagree, and the edge is the one that decides.
5. Watch the logs afterwards. A rule that is being honoured shows up as an absence of that user agent. Tracking AI bot activity covers how to read that and how to verify a bot is genuine.

Re-run the check quarterly. Vendors add tokens, the community lists change weekly, and a robots.txt written against last year's roster is silently out of date rather than broken in any way a WordPress dashboard would surface.

Checking AI crawler access with RankX AI

RankX AI is our AI visibility platform, and its AI crawler access checker is free and needs no account. It reads your robots.txt, resolves each rule against a roster of documented crawlers, and reports which are allowed and which are blocked, sorted by whether a crawler decides training or citations.

The website audit covers the same ground continuously for a site you own, alongside the rendering and structure checks that decide whether a crawler that is allowed in can use what it finds. Both exist because the answer to whether AI crawlers can read a WordPress site is measurable rather than a matter of opinion.