4xx status codes other than 429 have no effect on crawl rate.
Thing · Status code
4xx
The class of HTTP status codes for client errors; apart from 429 they do not slow Google's crawling.
- Claims
- 8
- In Google’s docs
- 1
- Said at the event
- 4
- Not in docs
- 0
- Kit items
- 3
Glossary · HTTP status code classes
1xx informational, 2xx success, 3xx redirection, 4xx client error and 5xx server error. Each class affects crawling differently, and 429 is the 4xx code that tells Google to slow down instead of saying there is nothing at the URL.
Google’s documentation 1
Documented in
Said at the event 4
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Consistent with docs 3
HTTP status codes fall into five classes, 1xx informational, 2xx success, 3xx redirection, 4xx client error and 5xx server error, and each class affects crawling differently.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Google has recently seen more 403 responses to its crawlers (described on stage as 'authentication required'); it treats them as client errors, technically equivalent to 404 or 410, and drops those pages from Search and its AI features.
Gary Illyes · Day 1 · How crawling errors affect Search
4xx responses do not affect a site's crawl budget, because Google expects pages, content and products to come and go as a natural part of the web.
Confirmed by docs 1
429 Too Many Requests is the exception among 4xx codes: instead of meaning there is nothing at the URL, it tells Google's crawler to slow down, and Google slows its crawling.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Press and analysis 3
AnalysisD1-C386
The stage point that 4xx responses do not affect crawl budget matches Google's documentation that 4xx codes have no effect on crawl rate (D1-C072), with one exception: 429 Too Many Requests counts as a server error and slows crawling like a 5xx (D1-C071, D1-C092).
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD1-C419
A 404 fetch is still a fetch: Google's 2017 crawl budget post says generally any URL Googlebot crawls counts towards a site's crawl budget, so the stage point that 4xx responses do not affect crawl budget is best read as 'they do not slow crawling, and a 404 tells Google to crawl that URL less over time'.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD2-C913
For URLs removed in a consolidation, return 404 or 410: Google's site move guide names those two codes, and Google's crawlers treat every 4xx code except 429 the same way, as content that does not exist, so the choice between them matters less than not redirecting to an unrelated page.
Ibrahim Anjro · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Built on these claims 3
Kit items about 4xx: their own words name it, or several of the claims they rest on do.
Developer requirements 2
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 2 of them naming 4xx; its own words name 4xx
Do not answer crawlers with 401, 402 or 403 on pages that should be in Search
Rests on 5 claims, 1 of them naming 4xx; its own words name 4xx
Also inglossary term HTTP status code classes
Connected things 13
Relations
- Includes 403 structure, no claim needed
- Includes 404 structure, no claim needed
- Includes 410 structure, no claim needed
- Includes 429 structure, no claim needed
Most often named with it
Things named in the same claim, with the number of claims they share.