Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
A soft 404 returns a success code while its main content looks like an error or an empty page; Google keeps it out of the index but keeps crawling it, which wastes crawl budget. On Day 2 Google explained that it detects soft 404s with a BERT-like language model trained on page structure and layout, so an error message alone in the main content makes a soft 404 while an error in the navigation need not, and simple keyword matching would not work (said at the event, not in Google's docs). Google listed four common causes that its troubleshooting guide also names (pages that look like errors, thin or empty content, server or CMS misconfiguration, JavaScript content that fails to load) and a fifth said only at the event, mistakes in Google's own systems, which it asked site owners to report in its forums. Single-page apps are a frequent source because the server returns 200 for every route, and Google documents a real 404, a JavaScript redirect to a URL that returns 404, or a noindex added with JavaScript as fixes. Soft 404s are also clustered together as duplicates, and index selection drops any that earlier steps missed. On Day 3 an audience member reported that on one large client site, pages clustered as soft 404s were still recrawled, but only every 160 to 190 days (one site's observation, not Google data). The second recording of Day 1 added the full definitions: Cherry Prommawin called a soft 404 a 404 in disguise, a 200 page whose content says 'not found', information the site should have sent as the status, and said soft 404s are still crawled like normal pages, with their bigger consequences in indexing; Gary Illyes called them one of the biggest problems on the internet right now for crawling and showing up in Search, and said a 200 only means the server believes it did what was asked. CDN captcha pages served with a 200 end up classified as soft 404s (Day 2 added that they can also be clustered as duplicates; Google's CDN post describes both outcomes), and Google's status code page says a 204 or an empty 2xx page also shows as a soft 404. Day 2's second recording confirmed that a client-rendered page that renders empty for Google is seen as thin content before it is treated as a soft 404, and Google's site move guide warns that redirecting many old URLs to an irrelevant page such as the home page might be treated as a soft 404. An audio recording of Day 1 added that soft 404s, like some other problem categories, are looked up in Search Console's page indexing report, while other crawl issues are debugged in the Crawl Stats report.
Things in this topic 21
Counts are claims that name the thing. All things
What to do
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-URL-08
An HTTP 200 OK status only means that the server believes it managed to do what the client asked for.
“just means that the server believes that it managed to accomplish whatever the user was asking for”
Wording checked against the slide or recording
Speaker Gary IllyesIn Day 1, 14:05 · How crawling worksEvidence transcript
Used byrequirement DEV-ERR-03
Soft 404s were named as a crawl problem alongside DNS and firewall issues, and described as one of the biggest problems on the internet right now for crawling and showing up in Search.
Speaker Gary IllyesIn Day 1, 14:35 · How crawling errors affect SearchEvidence notes, transcript
A 204 No Content response is not eligible for indexing, because the server confirms the request but returns no content to index.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-ERR-03
A soft 404 is a 404 in disguise: the page returns 200 but its content says something like 'page not found', information the site should have sent as the HTTP status.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-ERR-03
Soft 404 pages are still crawled like normal pages; their bigger consequences are for indexing, and they also affect crawling.
Speaker Cherry PrommawinIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.
Speaker Gary IllyesIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-SRV-02glossary term Soft 404
Google said soft 404s, like some other problem categories, are looked up in Search Console's page indexing report, while other crawl issues are debugged in the Crawl Stats report.
Speaker Gary IllyesIn Day 1, 14:35 · How crawling errors affect SearchEvidence transcript
Used byrequirement DEV-MON-03
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
Publisher Google, Google Search CentralAnnotates Day 1, 14:35 · How crawling errors affect Search
Used byrequirement DEV-ERR-01glossary term Soft 404
Google's status code page says that with a 204 No Content response Google receives no content and cannot process it, and that a 2xx page whose content is empty or an error message shows as a soft 404 in Search Console.
Publisher GoogleAnnotates Day 1, 14:35 · How crawling errors affect Search
Used byrequirement DEV-ERR-03
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
Author Ibrahim AnjroAnnotates Day 1, 14:35 · How crawling errors affect Search
Used byrequirements DEV-MON-04, DEV-SRV-02, DEV-SRV-03
Day 1 said a CDN captcha page served with HTTP 200 becomes a soft 404, while Day 2 (D2-C374, D2-C375) said such challenge pages are hard to recognise as errors and can be clustered as duplicates; Google's CDN post describes both outcomes, and in both the real pages drop out of Search.
Author Ibrahim AnjroAnnotates Day 1, 14:35 · How crawling errors affect Search
Google's slide defined a soft 404 in a JavaScript application as a page that serves a 'Not Found' message but returns a 200 HTTP status code.
“Your application serves a "Not Found" message but returns a 200 HTTP status code.”
Wording checked against the slide or recording
Speaker Erin SparlingIn Day 2, 11:15 · What is Google friendly JavaScriptEvidence slide photo, transcript
Used byrequirement DEV-ERR-02
Google listed four common causes of soft 404s: pages that look like errors but are not, thin or empty content, server or CMS misconfigurations, and JavaScript-dependent content that fails to load.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byrequirement DEV-ERR-03
Mistakes in Google's own systems are a further cause of soft 404s, and Gary Illyes asked site owners to report such mistakes in Google's forums.
“BONUS: Mistakes in Google's systems (that you should notify us about)”
Wording checked against the slide or recording
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
In single-page apps, a missing page often shows a custom 404 page while the server returns HTTP 200, because the front-end router, not the server, handles the 404.
Speaker Rebecca YuIn Day 2, 10:40 · Lightning session D: Rendering and JavaScriptEvidence transcript
Used byrequirement DEV-ERR-02glossary term Single-page app (SPA)
The fix for a client-side soft 404 is to make a missing page end with a real HTTP 404 status code instead of 200.
Speaker Rebecca YuIn Day 2, 10:40 · Lightning session D: Rendering and JavaScriptEvidence transcript
Used byrequirement DEV-ERR-02
A page that renders empty for Google, such as a client-side product page hit by these mistakes, is seen as thin content and ends up treated as a soft 404 even though users see a full page.
Speaker Rebecca YuIn Day 2, 10:40 · Lightning session D: Rendering and JavaScriptEvidence transcript
Used byrequirements DEV-ERR-03, DEV-REN-04
In JavaScript single-page apps, soft 404s typically arise because the server returns the app with a 200 status for every URL, so when the app shows a 'not found' message for a URL that does not exist, no error is reported.
Speaker Erin SparlingIn Day 2, 11:15 · What is Google friendly JavaScriptEvidence transcript
Used byrequirement DEV-ERR-02
The fix for a client-side soft 404 is to serve a real 404 error page where appropriate; how to detect URLs that do not exist depends on how the app works and where it is hosted.
Speaker Erin SparlingIn Day 2, 11:15 · What is Google friendly JavaScriptEvidence transcript
In an app routed with the History API, when a user reaches a page that does not exist, Erin Sparling suggested asking how they got there and redirecting to a real 404 page.
Speaker Erin SparlingIn Day 2, 11:15 · What is Google friendly JavaScriptEvidence transcript
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence 2 slide photos, transcript
Used byrequirements DEV-ERR-01, DEV-ERR-03glossary term Soft 404
Because error pages are worded in endless variations, of which 'page not found' is only the classic one, Google cannot detect soft 404s with simple error, word or keyword matching.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Google detects soft 404s with a language model, described as something like BERT, that is trained to understand the structure and layout of a page as well as its language, instead of reading the page as one flat wall of text.
“This is basically an LLM thing, something like BERT, that is specifically trained to understand page structure”
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
For soft 404 detection, the position of error text decides: an error in a less important part such as the navigation need not make a page a soft 404, but an error message alone in the main content, like 'Error establishing a database connection', does, much as a human would judge it.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byrequirement DEV-ERR-03
Google's soft 404 detection tells the page chrome, such as navigation and footer, apart from the main content by analysing the page's visual hierarchy alongside its text.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Gary Illyes said soft 404 detection keeps searchers from clicking into dead ends and avoids wasting site owners' resources on visitors sent to error pages.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Soft 404 pages are another reason Google clusters pages together, an outcome site owners usually do not want.
Speaker John MuellerIn Day 2, 11:55 · Handling web duplicationEvidence transcript
Used byrequirement DEV-ERR-01
A community speaker's team never redirected removed URLs to the homepage, even those that still brought traffic, because the aim was a clear signal about what the site is rather than keeping every visitor.
Speaker Martyna AğanoğluIn Day 2, 12:05 · Lightning session E: Managing Duplicates and Site MovesEvidence transcript
Used byrequirement DEV-CAN-10
A community speaker warned that HTTP 200 responses across a new domain show only that the URLs work, not that the content users came for is still there, so after launch the redirect map becomes the test.
Speaker David Carrasco PamiesIn Day 2, 12:05 · Lightning session E: Managing Duplicates and Site MovesEvidence transcript
Used byrequirement DEV-CAN-10
Index selection drops soft 404 pages that were not dropped earlier, for example when a document is reprocessed.
Speaker GoogleIn Day 2, 15:40 · Deciding what goes in the index?Evidence transcript
Google's JavaScript guides say that when client-side routing makes a real 404 status impractical, a single-page app can avoid soft 404s by redirecting with JavaScript to a URL whose server returns 404, or by adding a robots noindex meta tag with JavaScript.
Publisher Google Search CentralAnnotates Day 2, 10:40 · Lightning session D: Rendering and JavaScript
Used byrequirement DEV-ERR-02
For client-side rendered single-page apps, where meaningful status codes can be impossible or impractical, Google's documentation gives two ways to avoid soft 404s: a JavaScript redirect to a URL that returns a 404 status, or a noindex robots meta tag added with JavaScript.
Publisher Google Search CentralAnnotates Day 2, 11:15 · What is Google friendly JavaScript
Used byrequirement DEV-ERR-02
Google's site move guide says not to redirect many old URLs to one irrelevant destination such as the new site's home page, which can confuse users and might be treated as a soft 404, and to return a 404 or 410 for deleted or merged content that is not moved to the new site.
Publisher Google Search CentralAnnotates Day 2, 12:05 · Lightning session E: Managing Duplicates and Site Moves
Used byrequirement DEV-CAN-10
In a single-page app, let the server return 404 for unknown routes where possible; otherwise use one of the two client-side fixes Google documents for error views: a JavaScript redirect to a URL that returns 404, or a robots noindex added with JavaScript.
Author Ibrahim AnjroAnnotates Day 2, 10:40 · Lightning session D: Rendering and JavaScript
After moving to real paths, set the server or hosting rewrite rules so a direct request to every valid path returns 200 with the content (ideally server-rendered) and an unknown path returns a 404 status, which removes client-side soft 404s at the source.
Author Ibrahim AnjroAnnotates Day 2, 11:15 · What is Google friendly JavaScript
Used byrequirement DEV-ERR-02
Return real error status codes for error states, 404 or 410 for missing content and 503 for outages such as a failed database connection, also in single-page apps; a 200 page whose main content is only an error message is treated as a soft 404 even when header and navigation look normal.
Author Ibrahim AnjroAnnotates Day 2, 11:30 · Understanding what's on a page
Used byrequirements DEV-ERR-03, DEV-SRV-03
An audience member said that on the large site they observed, recrawl frequency followed the site hierarchy, demand and how well each page is internally linked.
From the audienceIn Day 3, 15:45 · How long does it take to..?Evidence transcript
With a typical refresh of about 30 days and deep or soft-404-like pages recrawled only every few months, link important deep pages from strong hub pages and fix pages that look like errors; then check recrawl intervals by site depth in server logs or the Crawl Stats report.
Author Ibrahim AnjroAnnotates Day 3, 15:45 · How long does it take to..?
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
Google said soft 404s, like some other problem categories, are looked up in Search Console's page indexing report, while other crawl issues are debugged in the Crawl Stats report.
CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.
Google said soft 404s, like some other problem categories, are looked up in Search Console's page indexing report, while other crawl issues are debugged in the Crawl Stats report.
Search Console's page indexing report breaks down the reasons why pages do or do not show in Search, and its categories help find patterns in how a site's content is crawled and served to Google's crawlers.
In single-page apps, a missing page often shows a custom 404 page while the server returns HTTP 200, because the front-end router, not the server, handles the 404.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
A page that renders empty for Google, such as a client-side product page hit by these mistakes, is seen as thin content and ends up treated as a soft 404 even though users see a full page.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
In JavaScript single-page apps, soft 404s typically arise because the server returns the app with a 200 status for every URL, so when the app shows a 'not found' message for a URL that does not exist, no error is reported.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
The fix for a client-side soft 404 is to serve a real 404 error page where appropriate; how to detect URLs that do not exist depends on how the app works and where it is hosted.
The fix for a client-side soft 404 is to make a missing page end with a real HTTP 404 status code instead of 200.
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
Soft 404s were named as a crawl problem alongside DNS and firewall issues, and described as one of the biggest problems on the internet right now for crawling and showing up in Search.
Google listed four common causes of soft 404s: pages that look like errors but are not, thin or empty content, server or CMS misconfigurations, and JavaScript-dependent content that fails to load.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
Google listed four common causes of soft 404s: pages that look like errors but are not, thin or empty content, server or CMS misconfigurations, and JavaScript-dependent content that fails to load.
A page that renders empty for Google, such as a client-side product page hit by these mistakes, is seen as thin content and ends up treated as a soft 404 even though users see a full page.
Google detects soft 404s with a language model, described as something like BERT, that is trained to understand the structure and layout of a page as well as its language, instead of reading the page as one flat wall of text.
BERT is used in indexing to understand each word in the context of the whole sentence rather than one word at a time.
For soft 404 detection, the position of error text decides: an error in a less important part such as the navigation need not make a page a soft 404, but an error message alone in the main content, like 'Error establishing a database connection', does, much as a human would judge it.
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
Soft 404 pages are another reason Google clusters pages together, an outcome site owners usually do not want.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.
CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.
Google's systems find it very hard to recognise a bot-challenge page as an error page, and Googlebot cannot solve the puzzles, such as selecting all the cats, that those pages ask for.
CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.
Google's Search Central post on CDNs says that if Google cannot recognise an error message served with a 200 status as an error, all pages showing the same message may be dropped from the index as duplicates, and recovery can be slow because Google has little incentive to recrawl duplicates. For bot-verification interstitials it recommends sending crawlers a 503 status.
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
Index selection drops soft 404 pages that were not dropped earlier, for example when a document is reprocessed.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
Index selection drops soft 404 pages that were not dropped earlier, for example when a document is reprocessed.
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
A community speaker warned that HTTP 200 responses across a new domain show only that the URLs work, not that the content users came for is still there, so after launch the redirect map becomes the test.
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
Google's slide defined a soft 404 in a JavaScript application as a page that serves a 'Not Found' message but returns a 200 HTTP status code.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
In JavaScript single-page apps, soft 404s typically arise because the server returns the app with a 200 status for every URL, so when the app shows a 'not found' message for a URL that does not exist, no error is reported.
In single-page apps, a missing page often shows a custom 404 page while the server returns HTTP 200, because the front-end router, not the server, handles the 404.
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.
A soft 404 is a page that should return an error status code but returns HTTP 200; from a crawling point of view it looks indexable, but because it has no real content Google throws it out of the index.
A soft 404 is a 404 in disguise: the page returns 200 but its content says something like 'page not found', information the site should have sent as the HTTP status.
Never serve error states, empty results or failed data loads with a 200 status
Rests on 10 claims, 9 of them in this topic
Return a real 404 for unknown routes in single-page apps, or use a documented client-side fallback
Rests on 7 claims, 7 of them in this topic
Redirect an old URL only to a page with the same intent, and return 404 or 410 when there is none
Rests on 7 claims, 3 of them in this topic
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 3 of them in this topic
Never serve bot challenges or error pages to crawlers with a 200 status
Rests on 7 claims, 2 of them in this topic
Answer planned maintenance and short outages with 503, for a day or two at most
Rests on 11 claims, 2 of them in this topic
Watch the Page indexing report and treat its not-indexed reasons as a work list
Rests on 10 claims, 1 of them in this topic
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 1 of them in this topic
Do not block JavaScript, CSS or API endpoints that pages need for rendering in robots.txt
Rests on 7 claims, 1 of them in this topic
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 1 of them in this topic