Google advises against using noindex to save crawl budget and against using robots.txt to temporarily reallocate budget. Use robots.txt only for pages you never want crawled, and 404 or 410 for removed pages.
Thing · Status code
410
The HTTP status code for a page that is gone for good; Google treats it much like a 404.
- Claims
- 7
- In Google’s docs
- 3
- Said at the event
- 2
- Not in docs
- 0
- Kit items
- 3
Google’s documentation 3
Documented in
Google's site move guide says not to redirect many old URLs to one irrelevant destination such as the new site's home page, which can confuse users and might be treated as a soft 404, and to return a 404 or 410 for deleted or merged content that is not moved to the new site.
Google Search Central · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Google's Removals tool help says a temporary removal request usually takes up to a day to process and lasts only about six months, so the content must also be removed permanently, for example with a 404 or 410, a noindex or a password.
Google Search Console Help · Day 3 · How long does it take to..?
Said at the event 2
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Consistent with docs 1
Google has recently seen more 403 responses to its crawlers (described on stage as 'authentication required'); it treats them as client errors, technically equivalent to 404 or 410, and drops those pages from Search and its AI features.
Gary Illyes · Day 1 · How crawling errors affect Search
Confirmed by docs 1
404 Not Found and 410 Gone both tell Google there is nothing at the URL, so the URL is not indexable.
Cherry Prommawin · Day 1 · How crawling errors affect Search
Press and analysis 2
AnalysisD2-C342
Return real error status codes for error states, 404 or 410 for missing content and 503 for outages such as a failed database connection, also in single-page apps; a 200 page whose main content is only an error message is treated as a soft 404 even when header and navigation look normal.
Ibrahim Anjro · Day 2 · Understanding what's on a page
AnalysisD2-C913
For URLs removed in a consolidation, return 404 or 410: Google's site move guide names those two codes, and Google's crawlers treat every 4xx code except 429 the same way, as content that does not exist, so the choice between them matters less than not redirecting to an unrelated page.
Ibrahim Anjro · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Built on these claims 3
Kit items about 410: their own words name it, or several of the claims they rest on do.
Developer requirements 3
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 2 of them naming 410; its own words name 410
Do not answer crawlers with 401, 402 or 403 on pages that should be in Search
Rests on 5 claims, 2 of them naming 410; its own words name 410
Redirect an old URL only to a page with the same intent, and return 404 or 410 when there is none
Rests on 7 claims, 1 of them naming 410; its own words name 410
Connected things 15
Relations
- Is a 4xx structure, no claim needed
Most often named with it
Things named in the same claim, with the number of claims they share.
- 404 7
- 4xx 2
- noindex 2
- Redirects 2
- Site move 2
- Soft 404 2
- 200 1
- 403 1
- 429 1
- 503 1
- Crawl budget 1
- Main content 1
- robots.txt 1
- Single-page app 1