Crawl budget / crawl economics

The finite compute Google will spend crawling and re-crawling a site, and the resource model that allocates it. Because Google “does not have infinite computing power,” it treats crawling as an investment decision: a domain earns sustained crawling only when its value justifies the cost. This is the supply-side gate beneath indexation — a page Google won’t spend budget to crawl can’t be indexed, and one it stops re-crawling gets dropped.

What governs the allocation

Per scaled-ai-content-crawl-economics, three factors set crawl allocation:

• Perceived inventory vs. actual utility — total URLs published is not what Google deems worth processing; publishing more pages does not auto-expand the budget.
• Demand — whether users and engines actually care about the topics.
• Popularity / staleness — baseline authority and link equity that justify the processing cost.

Burst-crawl → decay → de-index

The lifecycle that explains why programmatic content fades: a new site or section gets an initial burst-crawl and a freshness boost in visibility; without user signals it then decays past the quality threshold; pages not re-crawled within ~75–140 days risk removal from the index. Crucially, low-value clusters cause Google to reduce crawl frequency to that whole section, so decay compounds rather than levels off. (The specific day-window is the source author’s characterization, not a cited Google figure — see scaled-ai-content-crawl-economics caveats.)

Where it sits in the spoke

Crawl economics is the enforcement mechanism for the volume-hurts / density-over-volume thesis. Those pages argue thin pages lose in embedding space (a retrieval/demand-side model); crawl budget is the supply-side half — Google declines to crawl/index/retain them at all. It’s continuous with classic technical SEO: the same “don’t waste crawl on duplicates” logic in Google’s canonicalization guidance, generalized from duplicate URLs to whole thin-content archives, and the resource reading of the scaled content abuse policy.

Related

scaled-ai-content-crawl-economics · search-indexation · publishing-volume-hurts-seo · authority-density · google-consolidate-duplicate-urls · google-search-essentials