Crawl budget is the amount of crawling Google is willing to spend on your site in a given period — roughly, how many URLs Googlebot will fetch per day. For most small sites it never matters. For large sites it becomes the single factor that decides how much of your site ever gets into the index. Here is the plain-English version.
What crawl budget actually is
Google balances two things: crawl capacity (how hard it can hit your server without slowing it down) and crawl demand (how much it wants to crawl you, based on your site's popularity and freshness). The result is a practical daily limit on how many of your URLs get fetched.
When crawl budget does NOT matter
If your site has a few hundred or a few thousand pages, forget about crawl budget. Google can easily crawl a small site in full. Optimising crawl budget on a 200-page site is wasted effort — your problem is something else (usually content or authority).
When crawl budget DOES matter
Crawl budget becomes critical when you have tens of thousands of URLs or more, especially on a domain without much authority. Here is a real example: we launched a site with over 38,000 URLs and measured Google crawling only about 20 new URLs per day. At that rate, discovering the whole site would take years — so the vast majority sat in "Discovered – currently not indexed" limbo. The fix was not more pages; it was fewer. We pruned the site by 85%, down to its genuinely valuable pages, which concentrated crawl demand on URLs worth indexing.
What wastes crawl budget
- Low-value URLs — endless filter/sort parameters, faceted navigation, session IDs, near-duplicate pages.
- Redirect chains — every hop costs a fetch.
- Soft 404s and error pages — Googlebot fetching junk it will not index.
- Links to dead pages — crawling 404s and 410s instead of real content.
How to optimise crawl budget
- Prune ruthlessly. Fewer, better pages beat thousands of thin ones. But prune in steps and watch the impact — a sudden 85% cut can shock a site's serving before it recovers.
- Block low-value URLs in robots.txt or with parameters handling so Google spends its budget on pages that matter.
- Fix redirect chains and broken links.
- Improve internal linking so your important pages are shallow and well-connected.
- Build authority — crawl demand rises with your site's reputation.
How to see your own crawl budget
Search Console's Crawl Stats report shows how many requests Googlebot makes per day and the trend over time. One warning: if your site sits behind an edge cache — which is what a CDN like Cloudflare puts in front of your origin — your server logs will dramatically understate crawling, because most Googlebot hits are served from that cache and never reach your origin. We cover that trap in why Cloudflare hides most of Googlebot from your logs.
If you run a large site and suspect crawl budget is capping your indexing, our website audit measures your real crawl rate and shows exactly how much of your inventory Google is reaching.
