Google has seen an uptick in 402 Payment Required responses to its crawlers, an even more recent trend than the rise in 403 responses.
Gary Illyes · Day 1 · How crawling errors affect Search
Thing · Status code
403
The HTTP status code for a forbidden request; to Googlebot it is a client error and does not slow crawling the way 429 does.
- Claims
- 4
- In Google’s docs
- 0
- Said at the event
- 3
- Not in docs
- 1
- Kit items
- 2
Google’s documentation 0
Documented in
Said at the event 3
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Not in docs 1
Consistent with docs 2
A 403 response tells Google's crawler it has no permission to see the requested content, so there is nothing for Google to index.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Google has recently seen more 403 responses to its crawlers (described on stage as 'authentication required'); it treats them as client errors, technically equivalent to 404 or 410, and drops those pages from Search and its AI features.
Gary Illyes · Day 1 · How crawling errors affect Search
Press and analysis 1
AnalysisD1-C372
Do not answer verified Googlebot with 401, 402 or 403, for example from a login wall, a paywall or a pay-per-crawl setup: Google treats them like 404 and drops the pages, and Google's status code page says not to use 401 or 403 to limit crawling; to slow crawling temporarily, return 429 or 503.
Ibrahim Anjro · Day 1 · How crawling errors affect Search
Built on these claims 2
Kit items about 403: their own words name it, or several of the claims they rest on do.
Developer requirements 2
Do not answer crawlers with 401, 402 or 403 on pages that should be in Search
Rests on 5 claims, 3 of them naming 403; its own words name 403
Rests on 7 claims, 0 of them naming 403; its own words name 403
Connected things 7
Relations
- Is a 4xx structure, no claim needed
Most often named with it
Things named in the same claim, with the number of claims they share.