Google's crawl budget guide says each crawler has its own crawl demand, but the crawl capacity limit (hostload) is shared across all crawlers, so high demand from one crawler can reduce the capacity left for others.
Thing · Metric
Crawl demand
How much Google wants to crawl a site's URLs, driven by their popularity and how often they change.
- Claims
- 15
- In Google’s docs
- 1
- Said at the event
- 11
- Not in docs
- 2
- Kit items
- 6
Glossary · Crawl demand
How much Google wants to crawl a site, driven by its quality, how often its URLs change and how popular they are. Demand from other Google products, such as Ads, counts too, a Google panelist said.
Google’s documentation 1
Documented in
Said at the event 11
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Label
Grade
Day
Not in docs 2
The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual page.
Not in docsD3-C622
Google estimated that a crawl demand update driven by Search takes about 20 hours on average, with a minimum of minutes and an end point of weeks to months.
Consistent with docs 7
Consistent with docsD1-C090
Crawl budget is the combination of crawl rate limit and crawl demand.
Consistent with docsD1-C093
Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.
Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.
There is no easy way to tell whether a drop in Google's crawling comes from lower crawl demand or a lower crawl capacity limit; most of the time a capacity-limit drop is an abrupt step down.
A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.
When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.
Gary Illyes · Day 3 · How long does it take to..?
Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate covers only demand from Search.
Gary Illyes · Day 3 · How long does it take to..?
Confirmed by docs 1
If a site's server cannot cope with the extra crawling that higher crawl demand brings, Google reduces its crawling of that site again.
Nothing to verify 1
StageD1-C545
An audience member asked, for very large sites of about 100 million pages, what signs show that a site is limited by crawl budget, and how to tell a crawl capacity limit problem from a crawl demand problem.
From the audience · Day 1 · Q&A
Press and analysis 3
AnalysisD1-C067
A page with no internal links depends on sitemaps alone to be found, so it is discovered slowly and attracts little crawl demand.
Ibrahim Anjro · Day 1 · How crawling works
AnalysisD1-C095
New content inherits its starting crawl demand from the folder it sits in. Put new high-value content under sections Google already rates well, not under weak ones.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD1-C420
The stage remark that crawl demand follows the quality of the site as a whole sits beside the same talk's slide (D1-C094), which falls back to the parent path's aggregate only when a URL's own quality is unknown, and Google's crawl budget guide lists page quality among the demand factors; read it as site quality setting the baseline while known URL-level signals still count.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
Built on these claims 6
Kit items about Crawl demand: their own words name it, or several of the claims they rest on do.
Developer requirements 3
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 3 of them naming Crawl demand
Link every indexable page from the normal navigation or a hub page
Rests on 8 claims, 1 of them naming Crawl demand; its own words name Crawl demand
Rests on 4 claims, 1 of them naming Crawl demand; its own words name Crawl demand
Also inglossary terms Crawl demand, Crawl rate limit (hostload), Crawl budget
Connected things 4
Relations
- Part of Crawl budget structure, no claim needed
Most often named with it
Things named in the same claim, with the number of claims they share.