Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
Crawl budget is crawl rate limit combined with crawl demand. Google's crawl budget guide is written for sites with over 1 million unique pages changing weekly, over 10,000 pages changing daily, or many URLs reported as 'Discovered – currently not indexed'. On Day 2 Google called that status a crawl-scheduling state in which it knows a URL but does not want to crawl it yet (said at the event), while the Page indexing help explains it by expected server overload. A Google slide also questioned whether a new URL that fits a known duplicate URL pattern even needs to be crawled (not in Google's docs). Author’s view: rule out slow responses and server errors first, then treat slow crawling of a smaller site as a demand problem, which means quality. Day 3 gave the scale: Google said it knows hundreds of trillions of URLs (as of October 2026) and does not crawl them all, because a much smaller crawl space is enough, and its crawl chart said discovering a new URL can take weeks or never happen. The second recording of Day 1 added the crawl budget talk and the Q&A. Crawl budget is the finite resources Google allocates to crawling one site, which decide how many of its pages are discovered and how often they are revisited; Google crawls every URL that differs even when it leads to the same content, so infinite spaces such as calendars burn it, and the content worth improving or removing includes low-quality, spam, duplicate and soft error pages. Not every site needs to worry, and never did: a site under a few thousand URLs likely has no crawl budget problem, news sites are crawled aggressively anyway, and a higher crawl rate does not make a URL rank better. Every 2xx fetch consumes crawl budget, while 4xx responses were said not to affect it. To check for a problem Google pointed to the Crawl Stats report: whether total crawling has plateaued, whether the average response time limits Googlebot, the file-type breakdown for anomalies and the discovery versus refresh split. Log files alone cannot show inefficiency, because they do not reveal where the crawl limit lies, but they help on sites with filter parameters; many complaints turn out to come from a plugin, such as a WordPress calendar plugin that adds calendar parameters to the URLs of every page, and Google learns that such URLs are useless only from large samples; a panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with an Apache rule, which saves crawl budget for the URLs that matter (the exact mechanism is not clear in the recording). For news sites Gary Illyes advised measuring the time from publishing a URL to Googlebot's first crawl and investigating only a rising trend, adding that for fresh stories something like two hours is probably not great, while John Mueller said that if Crawl Stats shows about half of crawling going to discovering new pages, crawling of new pages is not the problem; Gary Illyes said disallowing a section such as /ads shifts crawl budget to the rest of the site. Author’s view: Google's guide says freed crawl budget shifts only on sites that already hit their capacity limit, a 404 is still a fetch that counts towards crawl budget, and a site between a few thousand URLs and Google's million-page threshold should check Crawl Stats before blaming crawl budget; for a consolidation of about 2,000 URLs the crawl-budget gain is likely minor.
Things in this topic 14
Counts are claims that name the thing. All things
What to do
Crawl budget is the combination of crawl rate limit and crawl demand.
“Crawl Budget = Crawl Rate Limit & Crawl Demand”
Wording checked against the slide or recording
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence 2 slide photos, transcript
Used byglossary term Crawl budget
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-URL-08
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence 2 slide photos, transcript
Used byrequirements DEV-SRV-08, DEV-URL-08
Soft 404 pages are still crawled like normal pages; their bigger consequences are for indexing, and they also affect crawling.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Crawl budget was described as the attention span Google gives a website, and better performance increases it.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence notes, transcript
Used byrequirement DEV-PRF-01glossary term Crawl budget
Crawl budget is the finite amount of resources Google allocates to crawling a specific website, and it determines how many of the site's pages are discovered and how often they are revisited.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byglossary term Crawl budget
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirements DEV-URL-08, DEV-URL-11
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
An audience member asked, in a question submitted before the event, whether crawl budget is still an SEO priority in 2026 or only relevant for very large sites.
From the audienceIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Not every site needs to worry about crawl budget, and this has always been so: a site with fewer than a few thousand URLs likely has no crawl budget problem, and a larger site does not necessarily have one either.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byglossary term Crawl budget
HTTP status codes do not all affect crawl budget in the same way, so knowing what each status code class does is one way to manage crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Every fetch that returns a 2xx success response consumes crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-SRV-08
4xx responses do not affect a site's crawl budget, because Google expects pages, content and products to come and go as a natural part of the web.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-ERR-01
An audience member asked, in questions submitted before the event, how often Googlebot should be expected to visit a site now that AI Mode has rolled out, and whether AI means more crawling.
From the audienceIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
An audience member asked, in a question submitted before the event, whether crawl frequency affects the ranking position of a URL.
From the audienceIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
A higher crawl rate does not make a URL rank better; it only means Google checks the site more often.
“if your crawl rate is increased, that doesn't mean that you would rank better.”
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
To check whether a site has a crawl budget problem, use Search Console's crawl report (Crawl Stats), which breaks crawl requests down by response and by file type and shows crawl problems Google finds.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-MON-04glossary term Crawl Stats report
The file-type breakdown of crawl requests in Search Console is useful for spotting anomalies, such as most crawling going to images on a site that has no images worth crawling.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-MON-04
Google advised checking crawl data in Search Console now and then without obsessing over crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Google follows robots.txt partly in its own interest: crawling an infinite URL space such as a calendar that robots.txt blocks would waste Google's crawling time as well as the site's resources.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
An audience member asked how to tell, from log files or by other methods, whether Google is spending crawl budget inefficiently on a site.
From the audienceIn Day 1, 16:35 · Q&AEvidence transcript
Log files alone cannot show crawl inefficiency because they do not reveal where Google's crawl limit for the site lies; many similar URLs being crawled may not matter if the site is nowhere near that limit.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-04
To judge crawl efficiency, check Search Console's Crawl Stats report: whether total crawling has plateaued over time, and whether the average response time shows the server is fast enough or is limiting Googlebot.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-04
Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
Many crawling complaints that reach Google's crawling team turn out to be caused by a plugin, for example a WordPress calendar plugin that adds calendar parameters to the URLs of every post, page, category and tag page.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
Google crawls new URLs, such as plugin-generated calendar pages, because it wants to see what they are; its automated systems learn that such URLs are useless only from large samples, which takes time.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
An audience member asked how a large news site can tell whether crawl budget is limiting how fast new articles are discovered (within minutes), and which statistics in Search Console and the logs show this.
From the audienceIn Day 1, 16:35 · Q&AEvidence transcript
Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
To check that new content is crawled fast enough, Gary Illyes advised measuring the time between publishing a URL and Googlebot's first crawl of it: ideally it stays flat or falls, and only a rising trend calls for investigation.
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
When new URLs are picked up more slowly, Gary Illyes advised checking the log files for URLs that Google crawls and recrawls a lot and judging whether they are useful.
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
John Mueller said the Crawl Stats report's split of crawling by purpose, discovery versus refresh, gives a rough view of whether discovery crawling is in a reasonable range, though measuring publish-to-first-crawl time is more accurate.
Speaker John MuellerIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
An audience member asked, for very large sites of about 100 million pages, what signs show that a site is limited by crawl budget, and how to tell a crawl capacity limit problem from a crawl demand problem.
From the audienceIn Day 1, 16:35 · Q&AEvidence transcript
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
Gary Illyes said that if the time from publishing to Googlebot's first crawl starts rising, how much it rises matters: for the crawling of fresh stories, something like two hours is probably not great (a smaller example figure is unclear in the recording).
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
John Mueller said that if the Crawl Stats report shows about half of a site's crawling going to discovering new pages, that is a lot of discovery crawling, which suggests that crawling of new pages is not the problem.
Speaker John MuellerIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-MON-11
Google's crawl budget guide is written for sites with over 1 million unique pages that change weekly, over 10,000 pages that change daily, or many URLs reported as 'Discovered – currently not indexed'.
Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget
On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
The stage rule of thumb (under a few thousand URLs crawl budget is unlikely to be a problem) and Google's crawl budget guide (D1-C097: sites with over a million pages that change about weekly, or over 10,000 pages that change daily) leave a middle range where a site should check Search Console's Crawl Stats report before blaming crawl budget for slow indexing.
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
A 404 fetch is still a fetch: Google's 2017 crawl budget post says generally any URL Googlebot crawls counts towards a site's crawl budget, so the stage point that 4xx responses do not affect crawl budget is best read as 'they do not slow crawling, and a 404 tells Google to crawl that URL less over time'.
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
Author Ibrahim AnjroAnnotates Day 1, 16:35 · Q&A
Google's slide on structurally similar content asked whether a new URL that fits a known duplicate pattern, such as /buy/seo-service, even needs to be crawled.
“Do we even need to crawl /buy/seo-service ?”
Wording checked against the slide or recording
Speaker John MuellerIn Day 2, 11:55 · Handling web duplicationEvidence slide photo
Used byrequirements DEV-CAN-08, DEV-URL-09
Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster loading and no crawl budget spent on content that no longer mattered.
Speaker Martyna AğanoğluIn Day 2, 12:05 · Lightning session E: Managing Duplicates and Site MovesEvidence transcript
'Discovered – currently not indexed' in Search Console is a crawl-scheduling state: Google knows the URL exists but does not want to crawl it yet. Of the two not-indexed statuses discussed, it was called the 'kind of nastier' one.
“The first one is kind of nastier.”
Speaker GoogleIn Day 2, 15:40 · Deciding what goes in the index?Evidence transcript
Used byrequirement DEV-MON-03glossary term Discovered – currently not indexed
Google's Page indexing report help says a 'Discovered – currently not indexed' page was found but not crawled yet, typically because Google wanted to crawl it but expected the crawl to overload the site, so it rescheduled the crawl.
Publisher Google Search Console HelpAnnotates Day 2, 15:40 · Deciding what goes in the index?
Used byrequirement DEV-MON-03glossary term Discovered – currently not indexed
Google's crawl budget guide is written for sites with over a million pages changing weekly or over 10,000 pages changing daily, so for a consolidation of about 2,000 URLs the crawl-budget gain is likely minor; the benefit of pruning such a site more plausibly comes from one strong URL per intent and consolidated signals.
Author Ibrahim AnjroAnnotates Day 2, 12:05 · Lightning session E: Managing Duplicates and Site Moves
The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.
Author Ibrahim AnjroAnnotates Day 2, 15:40 · Deciding what goes in the index?
Used byrequirement DEV-MON-03
Google's crawl chart says the discovery of a new URL can take weeks or never happen, because the web is huge.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence slide photo, transcript
Google said it knows hundreds of trillions of URLs (as of October 2026).
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Google does not crawl all the URLs it knows, because crawling everything would generally waste resources and a much smaller crawl space is enough.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Crawl budget is the finite amount of resources Google allocates to crawling a specific website, and it determines how many of the site's pages are discovered and how often they are revisited.
Crawl budget was described as the attention span Google gives a website, and better performance increases it.
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.
A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
Gary Illyes said that if the time from publishing to Googlebot's first crawl starts rising, how much it rises matters: for the crawling of fresh stories, something like two hours is probably not great (a smaller example figure is unclear in the recording).
John Mueller said that if the Crawl Stats report shows about half of a site's crawling going to discovering new pages, that is a lot of discovery crawling, which suggests that crawling of new pages is not the problem.
Google's slide on structurally similar content asked whether a new URL that fits a known duplicate pattern, such as /buy/seo-service, even needs to be crawled.
If the quality or popularity of a URL is unknown, the aggregate quality or popularity of its parent path is used, then that path's parent, and so on.
Google's slide on structurally similar content asked whether a new URL that fits a known duplicate pattern, such as /buy/seo-service, even needs to be crawled.
Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.
'Discovered – currently not indexed' in Search Console is a crawl-scheduling state: Google knows the URL exists but does not want to crawl it yet. Of the two not-indexed statuses discussed, it was called the 'kind of nastier' one.
The scheduler is shared infrastructure that decides what to fetch and when and sends URLs to the crawler. Each team decides the scheduling parameters for its own user agents.
'Discovered – currently not indexed' in Search Console is a crawl-scheduling state: Google knows the URL exists but does not want to crawl it yet. Of the two not-indexed statuses discussed, it was called the 'kind of nastier' one.
Google's crawl budget guide is written for sites with over 1 million unique pages that change weekly, over 10,000 pages that change daily, or many URLs reported as 'Discovered – currently not indexed'.
The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.
On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.
'Crawled – currently not indexed' is most of the time a quality issue rather than a technical one: the pages are usually low quality or useless for the index, for example duplicates or soft 404s.
On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.
Treat 'Crawled – currently not indexed' as a quality audit list: compare those URLs with indexed pages of the same type for thin, duplicate or soft-404-like content and for template differences before looking for technical faults.
On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.
A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster loading and no crawl budget spent on content that no longer mattered.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
Google said it knows hundreds of trillions of URLs (as of October 2026).
Google said there are trillions of URLs on the internet, or even more, that even Google cannot tell how many exist, and that some may never be discovered.
For news sites in particular, Google said the refresh of a known URL can literally take seconds (the reading 'news sites' is likely but not certain).
4xx responses do not affect a site's crawl budget, because Google expects pages, content and products to come and go as a natural part of the web.
4xx status codes other than 429 have no effect on crawl rate.
Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.
Not every site needs to worry about crawl budget, and this has always been so: a site with fewer than a few thousand URLs likely has no crawl budget problem, and a larger site does not necessarily have one either.
Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.
Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 9 of them in this topic
Measure the time from publishing a URL to Googlebot's first crawl of it
Rests on 7 claims, 6 of them in this topic
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 4 of them in this topic
Watch the Page indexing report and treat its not-indexed reasons as a work list
Rests on 10 claims, 3 of them in this topic
Support conditional requests with ETag or Last-Modified and answer 304 when nothing changed
Rests on 5 claims, 2 of them in this topic
Make pages that share a URL pattern differ in their main content
Rests on 6 claims, 1 of them in this topic
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 1 of them in this topic
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 1 of them in this topic
If facet URLs must be crawlable, use & separators, a fixed filter order and 404 for empty results
Rests on 5 claims, 1 of them in this topic
Compute the expected URL for every request and redirect variants to it
Rests on 8 claims, 1 of them in this topic