Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. For most small sites it is never the problem. For big or fast-changing sites, it decides how quickly new pages get found.
By Kristijan GjorgjioskiIndependent SEO consultantPublished 18 October 2026 · 7 min read
Crawl budget is a phrase that gets thrown around in audits far more than it deserves.
I see it flagged on 40-page brochure sites that Google crawls end to end every day. Meanwhile, the stores and directories that really do have a crawl budget problem often never hear the term.
This guide explains what crawl budget is, how to tell whether your site has a problem, and what to fix when it does.
Crawl budget is the set of URLs on a site that Googlebot can crawl and wants to crawl within a given time. It is set by two things: your server's crawl capacity limit and Google's crawl demand for your URLs.
Here's the key detail:
Google treats each hostname as its own site. So www.example.com and shop.example.com have separate crawl budgets.
This is how hard Googlebot can push your server without slowing it down.
Google describes it as the total time your server spends holding connections open for its crawlers. Fast, stable responses let the limit rise. Slow responses, 5xx errors and 429 "too many requests" codes make it fall.
This is how much Google wants to crawl your URLs.
It depends on how many URLs Google thinks you have, how popular those pages are and how stale its copy of them has become. A site migration can raise demand for a while, because every URL needs checking again.
Crawl budget is not indexing.
A crawled page can still be left out of the index.
It is also not a ranking signal.
Crawling your page more often does not make it rank higher.
And it is not something you can buy, request or switch on in Search Console.
Crawl budget matters when Google cannot keep up with the URLs you publish or change.
On most sites, it can.
Google's crawl budget guide for large sites puts rough numbers on it. The guidance is aimed at:
Google calls those figures a rough estimate, not exact thresholds.
In practice, these are the site types where I find real crawl problems:
| Site type | Why crawl budget becomes a problem |
|---|---|
| Online stores | Filters, sort orders and variants multiply one category into thousands of URLs |
| Directories and listings | Location x category pages, plus pagination on every combination |
| Publishers and job boards | Thousands of new or expiring URLs each week |
| Sites with JavaScript front ends | Rendering costs more, and broken links can generate endless URLs |
| Sites after a messy replatform | Redirect chains, old parameters and duplicate hostnames still live |
If you run a store, filters are almost always where the waste starts, which is why crawl control sits near the top of my technical SEO for e-commerce checks.
If your site has a few hundred pages and new ones get indexed within days, you do not have a crawl budget problem.
You might still have an indexing problem.
That is a different job, and it is part of what I check on any technical SEO engagement.
You diagnose it from Google's side, not from a crawler's health score. Three sources tell you most of what you need.
Open Search Console, then Settings, then Crawl stats.
According to Google's Crawl Stats help page, it shows total crawl requests, total download size, average response time and host status. You can also break requests down by response code, file type, purpose (discovery or refresh) and Googlebot type.
What I look for:
A growing pile of "Discovered - currently not indexed" URLs on a large site is the classic symptom. Google knows those URLs exist but has not got round to crawling them.
On a small site, the same status usually points to weak internal links or thin pages instead.
Logs show what Googlebot actually requested, URL by URL. They are the only way to see crawl spent on parameter URLs that never show up in Search Console.
When I get log access, the first question is simple:
What share of Googlebot's requests land on URLs that should not exist?
Most waste comes from URLs your site creates by accident.
Nobody plans thousands of filter pages. A theme or plugin generates them.
The usual offenders:
?sort=price and ?view=grid versions of the same list. Stores feel this the most. If you run one, my notes on Shopify duplicate content cover the platform's own version of the problem.
The fix is almost always subtraction.
Fewer junk URLs, faster responses, clearer signals.
Google's own best practices line up with what I do on large sites:
Merge duplicates where you can. Where you cannot, block low-value patterns such as alternative sort orders in robots.txt.
Then remove your internal links to those URLs, so Googlebot stops finding them.
This is the mistake I see most often.
A noindex tag tells Google not to index a page. But Google still has to request the page to read the tag, so the crawl is spent anyway. Google's guide says this directly.
Return 404 or 410 for pages that are gone for good.
Fix soft 404s so they stop being recrawled.
Google notes that blocked URLs stay in the crawl queue longer, so a clean 404 or 410 is the better signal for removed pages. That matters when you prune old content, which I cover in my content pruning guide.
List only canonical, indexable URLs that return 200. Use the lastmod date for pages that genuinely changed.
Point every redirect straight at its final URL. After a replatform, this is often the quickest win. A proper redirect map prevents chains forming in the first place.
Faster responses let Google crawl more in the same time. Google also recommends supporting HTTP 304 (Not Modified) responses, so unchanged pages cost less to recrawl.
Pro tip: If the URL Inspection tool reports "Hostload exceeded", Google is limited by your server, not by demand. That is the one case where more server resources directly mean more crawling.
These three get mixed up constantly, and the fix depends on which one is broken.
| Term | The question it answers | Typical fix |
|---|---|---|
| Crawl budget | Can Google fetch all the URLs it should? | Remove junk URLs, speed up the server |
| Rendering | Does Google see the content after JavaScript runs? | Server-side rendering or pre-rendering |
| Indexing | Does Google choose to keep the page? | Better content, canonicals, internal links |
A page can be crawled and not indexed.
It can be crawled, rendered and still ignored.
I once found a whole batch of new pages carrying a noindex tag by mistake. You can read how that played out in the language services case study. Crawl budget had nothing to do with it.
That is why I check crawling, rendering and indexing in order, rather than blaming the budget first. The full sequence is in my technical SEO checklist.
If your site is small, close this tab.
Spend the time on content and a sensible internal linking strategy instead.
If it is large, start with one question: how many crawlable URLs does your site generate that you never meant to publish? Answer that from logs or a full crawl, and the crawl budget work usually becomes obvious.
If you want a second pair of eyes on it, this kind of diagnosis is where my technical SEO work starts, and for a review across every discipline at once, an SEO audit is the better entry point.
Rarely. Google's own guidance says sites below roughly 10,000 pages that change daily, or a million pages overall, usually do not need to think about crawl budget. If new pages on a small site are slow to be indexed, the cause is more often weak internal linking, thin content or a noindex or robots.txt mistake than a shortage of crawl budget.
Open Search Console, go to Settings and choose Crawl stats. The report shows total crawl requests, download size, average response time and host status, plus breakdowns by response code, file type and crawl purpose. It is only available for root-level properties, such as a Domain property, and Google describes it as a report for advanced users.
Not directly. Blocking low-value URLs in robots.txt stops Googlebot spending time on them, but Google says it will not shift that time to other pages unless your site is already hitting its crawl capacity limit. Robots.txt is a way to stop waste, not a way to raise the overall amount of crawling.
No. Crawl budget affects whether and how quickly Google discovers and refreshes your pages, not where they rank. A page that is never crawled cannot rank at all, which is why crawl budget matters on large sites, but crawling more often does not push a page higher in the results.
About the author
Kristijan Gjorgjioski
Independent SEO consultant · 6+ years in SEO · 60+ client websites worked on
I'm an independent SEO consultant with 6+ years in SEO. I run audits, migrations and content programmes for businesses in Europe, the UK and the USA, and I build the automation behind my own research.
Book a free 30-minute call. I look at your site first and tell you plainly what I would fix, and in what order.