Crawl Budget
Last updated September 2026
Crawl budget is the number of pages a search engine will crawl on a site within a given timeframe, set by the site's crawl capacity limit and Google's crawl demand for it. For most small sites it is a non-issue; for large sites with thousands of pages, wasted crawl budget means new or updated content can sit undiscovered.
Crawl budget describes a real constraint: search engine crawlers do not have unlimited time or resources to spend on any one site, so they allocate a rough budget of how many pages to crawl and how often, based on how fast the server responds and how much crawling actually seems worthwhile for that site.
01
What determines crawl budget
Crawl rate limit is set by how much load a site's server can handle without slowing down or erroring for real visitors, so a slow or unstable server gets crawled more conservatively. Crawl demand reflects how much a search engine wants to crawl the site, driven by its overall popularity, how frequently content changes, and how much of the site consists of genuinely valuable, unique pages versus low-value bloat.
02
When crawl budget actually matters
For a typical local business site with a few dozen pages, crawl budget is essentially never the bottleneck, Google can crawl a small site fully with room to spare. It becomes a real concern for large e-commerce catalogs, large publishers, or any site with faceted navigation generating thousands of filter-combination URLs, where crawlers can burn their allotted budget on low-value pages and leave newer, more important content undiscovered for longer than it should be.
- 01
Faceted navigation generating near-infinite filter and sort URL combinations
- 02
Duplicate content across parameter variants
- 03
Soft 404s
pages that return a 200 status but show no real content
- 04
Slow server response times reducing the crawl rate limit overall
Takeaway
For a typical local business site with a few dozen pages, crawl budget is essentially never the bottleneck, Google can crawl a small site fully with room to spare.
03
Why it matters for agencies
Diagnosing crawl budget as the problem on a 40-page site wastes effort chasing a constraint that was never actually binding. Correctly identifying it on a large catalog or content-heavy site, and cleaning up the low-value URLs consuming the budget, prevents new content and product pages from sitting undiscovered for weeks. That triage is part of the technical due diligence Conduit runs before scaling a content or e-commerce SEO program for an agency's larger clients, drawing on Search Console's crawl stats and server logs.
