# Crawl Budget

> Source: https://rankxai.com/glossary/crawl-budget · Last updated: 2026-08-18

Crawl budget is the amount of crawling an engine will do on a site, set by what the server can take and by how much the engine wants the content. Crawl budget is a real constraint on very large sites and a non-issue on most others.

## Who actually has a crawl budget problem?

Very large sites, sites generating URLs faster than anybody can read them, and sites whose servers respond slowly enough to throttle the crawler. Google’s own guidance is that most sites do not need to think about this, and the advice industry has historically inverted that: crawl budget optimisation sold to a 200-page site is optimisation of something that was never scarce.

The two components are worth separating because they have different fixes. Capacity is about your server: if responses slow down, crawling slows down. Demand is about your content: pages nothing links to and nobody updates get crawled less because the engine has no reason to come back.

## What did AI crawlers change?

The volume, and who is paying for it. Cloudflare published the ratio behind its 1 July 2025 decision to [block AI crawlers by default](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/): Anthropic’s crawlers were fetching roughly 30,000 pages for every visitor sent back. That traffic is not a search engine paying for itself with referrals, and on a small origin it is a real cost.

The response is caching rather than blocking, for anyone who wants the visibility. A crawler served a cheap cached response costs almost nothing; the same crawler hitting an uncached, database-backed page thousands of times is a bill.

## Where does crawling actually get wasted?

Duplicate URLs from parameters, infinite calendars and filters, soft 404s returning a 200 status, and long redirect chains. Each of them consumes fetches on pages with no reason to exist, and the fix is structural rather than a setting: canonicalise, return real status codes, and stop generating addresses nobody asked for. The soft 404 is the worst of the four and the least noticed. A page returning 200 with “nothing found” on it is indistinguishable from a real page to a crawler, so it gets fetched, indexed and potentially retrieved, and on a large site it can generate thousands of them from one templating decision.

The fix is a status code rather than a message. A page with nothing on it should return 404 or 410, and an empty search result should usually not have a crawlable URL at all. Search Console reports soft 404s by name, which makes this one of the few crawl problems with a ready-made list rather than a diagnosis. It is worth checking that list quarterly on any site with search, filters or a calendar, because these appear from ordinary template changes rather than from anything anybody decided.

## Sources

- [Google Search Central: large site owner's guide to managing crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget), checked 2026-08-18
- [Cloudflare: Content Independence Day, 1 July 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), checked 2026-08-18