Crawl Budget

Near-duplicate pages, parameter URLs, large sets of thin location pages, and redirect chains can waste crawl capacity. Search engines may spend less time fetching important pages when they repeatedly encounter low-value or redundant URLs.

Google describes crawl allocation in terms of capacity and demand. Crawl capacity is how much Google can fetch without degrading the site; a slow or error-prone server lowers it. Crawl demand reflects factors such as perceived quality, freshness, and popularity. Technical changes can improve capacity, while content quality and site value affect demand.

A 200-page site is unlikely to exhaust its crawl budget. However, if many discovered URLs are near-duplicates or dead ends, Google may crawl the site less frequently. New content can then take longer to be discovered and indexed.

Use the Search Console crawl stats report and server logs to see which URLs Googlebot requested, how often it requested them, and what response it received. Logs may reveal frequent requests for filtered listing URLs, old redirects, or paginated archives that provide little value.

Fix recurring URL sources instead of blocking symptoms one at a time. Update faceted navigation, parameter handling, internal links, redirects, canonicals, and sitemaps so crawlers consistently discover the preferred URLs. Robots.txt can prevent crawling, but it does not automatically remove an already known URL from the index.

Review mass-produced pages for distinct value before publication. If Google inspects a large set and declines to index it, pause further releases. Publish a small batch, measure its indexation, and use the result to decide whether to publish more.