Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Thing · Metric
The number of URLs Google can and wants to crawl on a site, set by its crawl capacity limit and crawl demand.
Glossary · Crawl budget
How much Google crawls a site: the combination of crawl rate limit and crawl demand, the finite resources Google allocates to crawling one website. A site with fewer than a few thousand URLs likely has no crawl budget problem, Google said.
Documented in
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
Google, Google Search Central · Day 1 · How crawling errors affect Search
Google's crawl budget guide says each crawler has its own crawl demand, but the crawl capacity limit (hostload) is shared across all crawlers, so high demand from one crawler can reduce the capacity left for others.
Google's crawl budget guide is written for sites with over 1 million unique pages that change weekly, over 10,000 pages that change daily, or many URLs reported as 'Discovered – currently not indexed'.
Google advises against using noindex to save crawl budget and against using robots.txt to temporarily reallocate budget. Use robots.txt only for pages you never want crawled, and 404 or 410 for removed pages.
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Crawl budget is the combination of crawl rate limit and crawl demand.
Crawl budget was described as the attention span Google gives a website, and better performance increases it.
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
Crawl budget is the finite amount of resources Google allocates to crawling a specific website, and it determines how many of the site's pages are discovered and how often they are revisited.
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
HTTP status codes do not all affect crawl budget in the same way, so knowing what each status code class does is one way to manage crawl budget.
Every fetch that returns a 2xx success response consumes crawl budget.
4xx responses do not affect a site's crawl budget, because Google expects pages, content and products to come and go as a natural part of the web.
One fix for parameter variants of a URL is to work out the normalised URL and redirect the variants to it: the redirects cost crawl budget at first, but leave the site with a clean slate.
Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.
Gary Illyes · Day 1 · Q&A
Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.
Gary Illyes · Day 1 · Q&A
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.
A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster loading and no crawl budget spent on content that no longer mattered.
Martyna Ağanoğlu · Day 2 · Lightning session E: Managing Duplicates and Site Moves
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
The nofollow rule can still consume crawl budget: Google does not crawl through the nofollow link itself, but it still crawls the linked page when it finds that page through other links.
The non-standard crawl-delay rule is not processed by Googlebot, so it does not save any crawl budget.
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Not every site needs to worry about crawl budget, and this has always been so: a site with fewer than a few thousand URLs likely has no crawl budget problem, and a larger site does not necessarily have one either.
To check whether a site has a crawl budget problem, use Search Console's crawl report (Crawl Stats), which breaks crawl requests down by response and by file type and shows crawl problems Google finds.
An audience member asked, in a question submitted before the event, whether crawl budget is still an SEO priority in 2026 or only relevant for very large sites.
From the audience · Day 1 · How Google thinks about crawl budget
Google advised checking crawl data in Search Console now and then without obsessing over crawl budget.
An audience member asked how to tell, from log files or by other methods, whether Google is spending crawl budget inefficiently on a site.
From the audience · Day 1 · Q&A
An audience member asked how a large news site can tell whether crawl budget is limiting how fast new articles are discovered (within minutes), and which statistics in Search Console and the logs show this.
From the audience · Day 1 · Q&A
An audience member asked, for very large sites of about 100 million pages, what signs show that a site is limited by crawl budget, and how to tell a crawl capacity limit problem from a crawl demand problem.
From the audience · Day 1 · Q&A
The stage rule of thumb (under a few thousand URLs crawl budget is unlikely to be a problem) and Google's crawl budget guide (D1-C097: sites with over a million pages that change about weekly, or over 10,000 pages that change daily) leave a middle range where a site should check Search Console's Crawl Stats report before blaming crawl budget for slow indexing.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
The stage point that 4xx responses do not affect crawl budget matches Google's documentation that 4xx codes have no effect on crawl rate (D1-C072), with one exception: 429 Too Many Requests counts as a server error and slows crawling like a 5xx (D1-C071, D1-C092).
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
A 404 fetch is still a fetch: Google's 2017 crawl budget post says generally any URL Googlebot crawls counts towards a site's crawl budget, so the stage point that 4xx responses do not affect crawl budget is best read as 'they do not slow crawling, and a 404 tells Google to crawl that URL less over time'.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
The stage remark that crawl demand follows the quality of the site as a whole sits beside the same talk's slide (D1-C094), which falls back to the parent path's aggregate only when a URL's own quality is unknown, and Google's crawl budget guide lists page quality among the demand factors; read it as site quality setting the baseline while known URL-level signals still count.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
Ibrahim Anjro · Day 1 · Q&A
Google's crawl budget guide does not express the crawl capacity limit in requests per second: it limits the total time a server spends holding connections open for Google, counting both the number of parallel connections and their duration. That fits the panel's advice to watch how many connections Googlebot opens (D1-C494): fewer connections is the documented form of a lower capacity limit.
Ibrahim Anjro · Day 1 · Q&A
Google's crawl budget guide is written for sites with over a million pages changing weekly or over 10,000 pages changing daily, so for a consolidation of about 2,000 URLs the crawl-budget gain is likely minor; the benefit of pruning such a site more plausibly comes from one strong URL per intent and consolidated signals.
Ibrahim Anjro · Day 2 · Lightning session E: Managing Duplicates and Site Moves
The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.
Ibrahim Anjro · Day 2 · Deciding what goes in the index?
Kit items about Crawl budget: their own words name it, or several of the claims they rest on do.
Do not use noindex, nofollow or crawl-delay to save crawl budget
Rests on 6 claims, 5 of them naming Crawl budget; its own words name Crawl budget
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 5 of them naming Crawl budget; its own words name Crawl budget
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 3 of them naming Crawl budget; its own words name Crawl budget
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 3 of them naming Crawl budget; its own words name Crawl budget
Support conditional requests with ETag or Last-Modified and answer 304 when nothing changed
Rests on 5 claims, 2 of them naming Crawl budget; its own words name Crawl budget
Compute the expected URL for every request and redirect variants to it
Rests on 8 claims, 2 of them naming Crawl budget
Measure the time from publishing a URL to Googlebot's first crawl of it
Rests on 7 claims, 1 of them naming Crawl budget; its own words name Crawl budget
Rests on 20 claims, 1 of them naming Crawl budget; its own words name Crawl budget
Rests on 2 claims, 2 of them naming Crawl budget; its own words name Crawl budget
Also inglossary term Crawl budget
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
To check whether a site has a crawl budget problem, use Search Console's crawl report (Crawl Stats), which breaks crawl requests down by response and by file type and shows crawl problems Google finds.
Things named in the same claim, with the number of claims they share.