Crawl Budget

Who actually has a crawl budget problem?

Very large sites, sites generating URLs faster than anybody can read them, and sites whose servers respond slowly enough to throttle the crawler. Google’s own guidance is that most sites do not need to think about this, and the advice industry has historically inverted that: crawl budget optimisation sold to a 200-page site is optimisation of something that was never scarce.

The two components are worth separating because they have different fixes. Capacity is about your server: if responses slow down, crawling slows down. Demand is about your content: pages nothing links to and nobody updates get crawled less because the engine has no reason to come back.

What did AI crawlers change?

The volume, and who is paying for it. Cloudflare published the ratio behind its 1 July 2025 decision to block AI crawlers by default: Anthropic’s crawlers were fetching roughly 30,000 pages for every visitor sent back. That traffic is not a search engine paying for itself with referrals, and on a small origin it is a real cost.

The response is caching rather than blocking, for anyone who wants the visibility. A crawler served a cheap cached response costs almost nothing; the same crawler hitting an uncached, database-backed page thousands of times is a bill.

Where does crawling actually get wasted?

Duplicate URLs from parameters, infinite calendars and filters, soft 404s returning a 200 status, and long redirect chains. Each of them consumes fetches on pages with no reason to exist, and the fix is structural rather than a setting: canonicalise, return real status codes, and stop generating addresses nobody asked for. The soft 404 is the worst of the four and the least noticed. A page returning 200 with “nothing found” on it is indistinguishable from a real page to a crawler, so it gets fetched, indexed and potentially retrieved, and on a large site it can generate thousands of them from one templating decision.

The fix is a status code rather than a message. A page with nothing on it should return 404 or 410, and an empty search result should usually not have a crawlable URL at all. Search Console reports soft 404s by name, which makes this one of the few crawl problems with a ready-made list rather than a diagnosis. It is worth checking that list quarterly on any site with search, filters or a calendar, because these appear from ordinary template changes rather than from anything anybody decided.

Related terms

• Robots.txtRobots.txt is a file at a site’s root telling automated crawlers which paths they may fetch.
• XML SitemapAn XML sitemap is a file listing the URLs on a site that you want search engines to know about, optionally with the date each was last modified.
• Canonical TagA canonical tag is a link element naming the preferred URL for a page when the same content is reachable at more than one address.
• AI CrawlerAn AI crawler is an automated fetcher run by an AI company to collect web pages.