Google treats network timeouts, connection resets and DNS errors like 5xx server errors: crawling slows down immediately, and already indexed URLs that stay unreachable are removed from Google's index within days.
Thing · Status code
5xx
The class of HTTP status codes for server errors; Google slows crawling when it sees them and drops URLs that keep returning them.
- Claims
- 12
- In Google’s docs
- 4
- Said at the event
- 4
- Not in docs
- 0
- Kit items
- 8
Glossary · HTTP status code classes
1xx informational, 2xx success, 3xx redirection, 4xx client error and 5xx server error. Each class affects crawling differently, and 429 is the 4xx code that tells Google to slow down instead of saying there is nothing at the URL.
Google’s documentation 4
Documented in
5xx and 429 responses prompt Google's crawlers to slow down temporarily. Already indexed URLs are preserved in the index for a while but eventually dropped.
If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the cached copy for up to 30 days.
Chrome's Lighthouse documentation lists an llms.txt audit among its agentic browsing audits: it flags a server error when llms.txt is fetched and marks the audit not applicable when the file is missing, because providing the file is optional for now.
Chrome for Developers · Day 1 · Q&A
Said at the event 4
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Consistent with docs 3
HTTP status codes fall into five classes, 1xx informational, 2xx success, 3xx redirection, 4xx client error and 5xx server error, and each class affects crawling differently.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Consistent with docsD1-C092
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Consistent with docsD1-C099
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
Confirmed by docs 1
Google slows crawling when a site returns 5xx errors, because a 5xx usually means the server, and often the whole site, cannot serve requests, and Google does not want to break the site.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Press and analysis 4
AnalysisD1-C386
The stage point that 4xx responses do not affect crawl budget matches Google's documentation that 4xx codes have no effect on crawl rate (D1-C072), with one exception: 429 Too Many Requests counts as a server error and slows crawling like a 5xx (D1-C071, D1-C092).
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD2-C210
An API or CDN on its own host needs its own robots.txt check: a blanket Disallow there, or a robots.txt that returns 5xx errors, can stop Google fetching the data a page renders from.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
AnalysisD2-C708
The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.
Ibrahim Anjro · Day 2 · Deciding what goes in the index?
AnalysisD3-C621
Spoken and slide figures for capacity increases differ (one to three weeks, up to a month, versus 1-2 weeks typical and 1-3 weeks in recovery), but the lesson is the same: a burst of 5xx errors cuts crawling within hours and recovery takes weeks, so keep servers stable before launches and migrations.
Ibrahim Anjro · Day 3 · How long does it take to..?
Built on these claims 8
Kit items about 5xx: their own words name it, or several of the claims they rest on do.
Developer requirements 5
Answer planned maintenance and short outages with 503, for a day or two at most
Rests on 11 claims, 4 of them naming 5xx; its own words name 5xx
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 2 of them naming 5xx; its own words name 5xx
Keep robots.txt reachable on every host: 200 with rules, or 404 if there are none
Rests on 4 claims, 2 of them naming 5xx; its own words name 5xx
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 1 of them naming 5xx; its own words name 5xx
Let verified Google crawlers through firewalls, CDNs and bot protection
Rests on 11 claims, 1 of them naming 5xx; its own words name 5xx
Facts 1
- DocumentedF-017
Rests on 1 claim, 1 of them naming 5xx; its own words name 5xx
Also inglossary terms Crawl rate limit (hostload), HTTP status code classes
Connected things 12
Relations
- Includes 503 structure, no claim needed
- Affects Crawl capacity limit
1 claim, 1 documented
- Consistent with docsD1-C092
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Most often named with it
Things named in the same claim, with the number of claims they share.
- 429 3
- Crawl budget 3
- 4xx 2
- robots.txt 2
- CDN 1
- Chrome 1
- Crawl capacity limit 1
- Disallow 1
- llms.txt 1
- Redirects 1