Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
Hostload, the crawl capacity limit, is a host-wide limit shared by all Google crawlers, so heavy crawling by one Google product can leave less capacity for the others. It falls when connect time or time to first byte rise, or when the server returns 429, 5xx or network errors. On Day 2 a Google slide showed an example fetch record, as passed on to processing, that includes connect time and time to first byte in milliseconds (not in Google's docs). Day 3 gave timings: when a site starts serving 500 errors, Google lowers its crawl capacity within about four hours on average, while increases take one to three weeks because Google first needs to know that higher demand will last, and a continuously running process recalculates capacity within about a month (the spoken and slide figures differ slightly and are not in Google's docs). Google's crawl rate guide confirms that many 500, 503 or 429 responses reduce the crawl rate, which recovers automatically once the errors drop. The second recording of Day 1 added that crawl rate limit, or hostload, is a proxy for how many requests per second the connection to a site can handle, and that it applies per host, not necessarily per site: a site's CDN, app, subdomains and www host may be different hosts. Cherry Prommawin said 429 is the one 4xx code that tells Google to slow down, and that a 5xx makes Google slow down so as not to break the site; a rise in errors can come from a CDN throttling crawlers by injecting 429 or 503 responses on the network path. In the Q&A Google said that when higher demand brings more crawling than a server can cope with, Google reduces it again, and, asked how a site of about 100 million pages can tell a crawl capacity problem from a crawl demand problem, that a capacity drop is most of the time an abrupt step down, illustrated as hostload falling from 10 to 5, which a panelist said means up to five requests per second, and that a fall in the number of connections Googlebot opens shows the capacity limit changed. Google added that demand from Search, Google Ads or another Google product raises crawling only as far as the site's crawl capacity allows. Author’s view: Google's crawl budget guide does not express the capacity limit in requests per second but as the total time a server spends holding connections open for Google, counting parallel connections and their duration, so fewer connections is the documented form of a lower limit. Author’s view: to slow crawling for a short time return 429 or 503, never 401, 402 or 403, which Google treats like 404.
Things in this topic 11
Counts are claims that name the thing. All things
What to do
Google Search, Ads, Shopping and Images all request through one centralised crawling infrastructure, whose primary mandate is to fetch from the internet while strictly preventing the overloading of external servers.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-PRF-01
Crawl rate limit, or hostload, is a host-wide metric shared across all Google crawlers.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirements DEV-MON-04, DEV-PRF-01glossary term Crawl rate limit (hostload)
429 Too Many Requests is the exception among 4xx codes: instead of meaning there is nothing at the URL, it tells Google's crawler to slow down, and Google slows its crawling.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-SRV-03glossary term HTTP status code classes
Google slows crawling when a site returns 5xx errors, because a 5xx usually means the server, and often the whole site, cannot serve requests, and Google does not want to break the site.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-SRV-03
A rise in HTTP errors in crawl reports can come from a CDN throttling crawlers by injecting 429 or 503 responses on the network path between Google's crawler and the site.
Speaker Gary IllyesIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-SRV-02
Crawl rate limit, or hostload, is a proxy for how many requests per second the connection to a site can handle before Google hits the limit.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-PRF-01
Hostload applies per host, not necessarily per site: depending on how a site is configured, its CDN, its app, its subdomains and its main www host may be different hosts.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)
Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
If a site's server cannot cope with the extra crawling that higher crawl demand brings, Google reduces its crawling of that site again.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-PRF-01
An audience member asked, for very large sites of about 100 million pages, what signs show that a site is limited by crawl budget, and how to tell a crawl capacity limit problem from a crawl demand problem.
From the audienceIn Day 1, 16:35 · Q&AEvidence transcript
There is no easy way to tell whether a drop in Google's crawling comes from lower crawl demand or a lower crawl capacity limit; most of the time a capacity-limit drop is an abrupt step down.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)
To check whether Google lowered a site's crawl capacity limit, look at the number of connections Googlebot opens to the site: if it has dropped, the capacity limit changed.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)
A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byglossary term Crawl demand
A Google panelist illustrated a crawl capacity limit drop as a step down in the site's hostload, for example from 10 to 5, which the panelist said means Googlebot then makes up to five requests per second (the recording adds 'per connection', which is unclear).
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Google treats network timeouts, connection resets and DNS errors like 5xx server errors: crawling slows down immediately, and already indexed URLs that stay unreachable are removed from Google's index within days.
“Google treats network timeouts, connection reset, and DNS errors similarly to 5xx server errors.”
Publisher GoogleAnnotates Day 1, 14:35 · How crawling errors affect Search
Used byrequirements DEV-MON-04, DEV-SRV-01, DEV-SRV-03fact F-017
5xx and 429 responses prompt Google's crawlers to slow down temporarily. Already indexed URLs are preserved in the index for a while but eventually dropped.
“already indexed URLs are preserved in the index, but eventually dropped”
Publisher GoogleAnnotates Day 1, 14:35 · How crawling errors affect Search
Used byrequirement DEV-SRV-03
Google's crawl budget guide says each crawler has its own crawl demand, but the crawl capacity limit (hostload) is shared across all crawlers, so high demand from one crawler can reduce the capacity left for others.
“high demand from one crawler can reduce the capacity available for others”
Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)
Because every Google product shares the same host capacity, Ads or Shopping fetches that hit a slow server can reduce how much is crawled for Search.
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Do not answer verified Googlebot with 401, 402 or 403, for example from a login wall, a paywall or a pay-per-crawl setup: Google treats them like 404 and drops the pages, and Google's status code page says not to use 401 or 403 to limit crawling; to slow crawling temporarily, return 429 or 503.
Author Ibrahim AnjroAnnotates Day 1, 14:35 · How crawling errors affect Search
The stage point that 4xx responses do not affect crawl budget matches Google's documentation that 4xx codes have no effect on crawl rate (D1-C072), with one exception: 429 Too Many Requests counts as a server error and slows crawling like a 5xx (D1-C071, D1-C092).
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Google's crawl budget guide does not express the crawl capacity limit in requests per second: it limits the total time a server spends holding connections open for Google, counting both the number of parallel connections and their duration. That fits the panel's advice to watch how many connections Googlebot opens (D1-C494): fewer connections is the documented form of a lower capacity limit.
Author Ibrahim AnjroAnnotates Day 1, 16:35 · Q&A
Used byrequirement DEV-PRF-01
A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.
Speaker Cherry PrommawinIn Day 2, 10:25 · How is HTML interpretedEvidence slide photo
Used byrequirement DEV-PRF-01
Google's crawl chart puts a crawl capacity update at seconds when backing off, typically 4 hours or 1-2 weeks, and 1-3 weeks in recovery.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence slide photo, transcript
When a site starts serving 500 errors, Google lowers the crawl capacity allocated to the site within about four hours on average.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Used byrequirement DEV-SRV-03
Increases in crawl capacity take longer than decreases, within one to three weeks, because Google first needs to know that the higher demand will last.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Used byrequirement DEV-SRV-03
A continuously running process recalculates each site's crawl capacity within a month, so a capacity change takes up to about a month at most.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Google's crawl rate guide says that when a significant number of URLs return 500, 503 or 429, Google reduces the site's crawl rate, which starts increasing again automatically once the errors drop; it warns against doing this for longer than 1-2 days.
Publisher GoogleAnnotates Day 3, 15:45 · How long does it take to..?
Used byrequirement DEV-SRV-03
Spoken and slide figures for capacity increases differ (one to three weeks, up to a month, versus 1-2 weeks typical and 1-3 weeks in recovery), but the lesson is the same: a burst of 5xx errors cuts crawling within hours and recovery takes weeks, so keep servers stable before launches and migrations.
Author Ibrahim AnjroAnnotates Day 3, 15:45 · How long does it take to..?
Used byrequirement DEV-SRV-03
Google Search, Ads, Shopping and Images all request through one centralised crawling infrastructure, whose primary mandate is to fetch from the internet while strictly preventing the overloading of external servers.
Google does not let each team build its own crawler: its many crawlers share one crawler infrastructure, because every crawler must accomplish a few specific tasks and obey Google's internal crawling policies.
Crawl rate limit, or hostload, is a proxy for how many requests per second the connection to a site can handle before Google hits the limit.
Crawl rate limit, or hostload, is a host-wide metric shared across all Google crawlers.
Googlebot as a single standalone crawler is a historical idea: Google crawls through a centralised crawling infrastructure, so a request from a Google user agent in server logs is a request routed through that shared platform.
Google Search, Ads, Shopping and Images all request through one centralised crawling infrastructure, whose primary mandate is to fetch from the internet while strictly preventing the overloading of external servers.
Hostload applies per host, not necessarily per site: depending on how a site is configured, its CDN, its app, its subdomains and its main www host may be different hosts.
Crawl rate limit, or hostload, is a host-wide metric shared across all Google crawlers.
Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.
Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.
A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.
Different Google teams prioritise crawling differently: web search cares a lot about the quality of a site and its content, while Ads wants to check every publisher page that wants to appear in Google Ads, so it schedules those URLs as they come in.
A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.
A Google panelist illustrated a crawl capacity limit drop as a step down in the site's hostload, for example from 10 to 5, which the panelist said means Googlebot then makes up to five requests per second (the recording adds 'per connection', which is unclear).
Crawl rate limit, or hostload, is a proxy for how many requests per second the connection to a site can handle before Google hits the limit.
Google's crawl budget guide does not express the crawl capacity limit in requests per second: it limits the total time a server spends holding connections open for Google, counting both the number of parallel connections and their duration. That fits the panel's advice to watch how many connections Googlebot opens (D1-C494): fewer connections is the documented form of a lower capacity limit.
A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.
The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.
A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.
The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.
A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Google's crawl chart puts a crawl capacity update at seconds when backing off, typically 4 hours or 1-2 weeks, and 1-3 weeks in recovery.
When a site starts serving 500 errors, Google lowers the crawl capacity allocated to the site within about four hours on average.
Google treats network timeouts, connection resets and DNS errors like 5xx server errors: crawling slows down immediately, and already indexed URLs that stay unreachable are removed from Google's index within days.
When a site starts serving 500 errors, Google lowers the crawl capacity allocated to the site within about four hours on average.
Google slows crawling when a site returns 5xx errors, because a 5xx usually means the server, and often the whole site, cannot serve requests, and Google does not want to break the site.
When a site starts serving 500 errors, Google lowers the crawl capacity allocated to the site within about four hours on average.
When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.
Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate covers only demand from Search.
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 11 of them in this topic
Answer planned maintenance and short outages with 503, for a day or two at most
Rests on 11 claims, 8 of them in this topic
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 2 of them in this topic
Let verified Google crawlers through firewalls, CDNs and bot protection
Rests on 11 claims, 1 of them in this topic
Never serve bot challenges or error pages to crawlers with a 200 status
Rests on 7 claims, 1 of them in this topic
Rests on 1 claim, 1 of them in this topic