5xx and 429 responses prompt Google's crawlers to slow down temporarily. Already indexed URLs are preserved in the index for a while but eventually dropped.
Thing · Status code
429
The HTTP status code for too many requests; Google treats it like a server error and slows crawling.
- Claims
- 10
- In Google’s docs
- 3
- Said at the event
- 3
- Not in docs
- 0
- Kit items
- 8
Google’s documentation 3
Documented in
4xx status codes other than 429 have no effect on crawl rate.
Google's crawl rate guide says that when a significant number of URLs return 500, 503 or 429, Google reduces the site's crawl rate, which starts increasing again automatically once the errors drop; it warns against doing this for longer than 1-2 days.
Google · Day 3 · How long does it take to..?
Said at the event 3
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Consistent with docs 2
A rise in HTTP errors in crawl reports can come from a CDN throttling crawlers by injecting 429 or 503 responses on the network path between Google's crawler and the site.
Gary Illyes · Day 1 · How crawling errors affect Search
Consistent with docsD1-C092
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Confirmed by docs 1
429 Too Many Requests is the exception among 4xx codes: instead of meaning there is nothing at the URL, it tells Google's crawler to slow down, and Google slows its crawling.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Press and analysis 4
AnalysisD1-C078
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
Ibrahim Anjro · Day 1 · How crawling errors affect Search
AnalysisD1-C372
Do not answer verified Googlebot with 401, 402 or 403, for example from a login wall, a paywall or a pay-per-crawl setup: Google treats them like 404 and drops the pages, and Google's status code page says not to use 401 or 403 to limit crawling; to slow crawling temporarily, return 429 or 503.
Ibrahim Anjro · Day 1 · How crawling errors affect Search
AnalysisD1-C386
The stage point that 4xx responses do not affect crawl budget matches Google's documentation that 4xx codes have no effect on crawl rate (D1-C072), with one exception: 429 Too Many Requests counts as a server error and slows crawling like a 5xx (D1-C071, D1-C092).
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD2-C913
For URLs removed in a consolidation, return 404 or 410: Google's site move guide names those two codes, and Google's crawlers treat every 4xx code except 429 the same way, as content that does not exist, so the choice between them matters less than not redirecting to an unrelated page.
Ibrahim Anjro · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Built on these claims 8
Kit items about 429: their own words name it, or several of the claims they rest on do.
Developer requirements 6
Answer planned maintenance and short outages with 503, for a day or two at most
Rests on 11 claims, 4 of them naming 429; its own words name 429
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 2 of them naming 429; its own words name 429
Never serve bot challenges or error pages to crawlers with a 200 status
Rests on 7 claims, 2 of them naming 429
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 1 of them naming 429; its own words name 429
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 1 of them naming 429; its own words name 429
1 more
Do not answer crawlers with 401, 402 or 403 on pages that should be in Search
Rests on 5 claims, 0 of them naming 429; its own words name 429
Also inglossary terms Crawl rate limit (hostload), HTTP status code classes
Connected things 16
Relations
- Is a 4xx structure, no claim needed
- Affects Crawl capacity limit
1 claim, 1 documented
- Consistent with docsD1-C092
Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.
Most often named with it
Things named in the same claim, with the number of claims they share.
- 4xx 4
- 503 4
- 5xx 3
- 404 2
- Googlebot 2
- 200 1
- 403 1
- 410 1
- CDN 1
- Crawl budget 1
- Crawl capacity limit 1
- Redirects 1
- Site move 1
- Soft 404 1