Crawl Budget

Budget problems are usually self-inflicted: near-duplicate pages, parameter URLs, thin location pages by the hundred, redirect chains. The crawler spends its visit on junk and returns less often for the pages that matter.

Google describes the allocation as two forces. Crawl capacity is how hard a site can be fetched without degrading — a slow or error-prone server lowers it directly. Crawl demand is how much Google wants the site, driven by perceived quality, freshness, and popularity. Capacity is an engineering problem. Demand is an editorial one, and it is the side most sites lose.

The reason it matters below the enterprise scale is not exhaustion but signalling. A 200-page site will never run out of crawl budget in the literal sense. But a site where a third of discovered URLs turn out to be near-duplicates or dead ends is teaching Google something about the average value of fetching it, and that lesson gets applied to the pages you care about. The cost shows up as new content taking weeks to index rather than days.

Diagnosis lives in the Search Console crawl stats report and, more precisely, in server logs: which URLs Googlebot actually requested, how often, and what it received. Log data routinely shows crawlers spending most of their visits on filtered listing URLs, old redirects, and paginated archives nobody has looked at in years.

This is a core reason mass-produced pages must clear a value bar before publication. A thousand pages Google inspects and declines to index are not neutral — they are evidence about the site. We gate programmatic batches on measured indexation of the previous batch for exactly this reason: publish thirty, read the verdict in indexation, then earn the next fifty.