An audience member asked whether HTML sitemaps still matter for large sites, whether they improve Googlebot's crawl discovery and whether they pass link equity to deeper pages.
From the audienceEvidence slide photo
Things
Day 2 · Thursday 1 October 2026 · 10:15
Welcome to indexing day!
Presented by Gary Illyes and Cherry Prommawin (named in the host's hand-over; turns not attributable). Highly rated Slido questions skipped on Day 1 for lack of time: four on slides with written answers, one more (sloppy migrations) only spoken, then a recap of Day 1. Transcript from a second attendee recording.
Shown on screen 20
QQuestion and answers
Not in docsD2-C002
Google answered on a Q&A slide that good HTML sitemaps are hard to make for large sites.
“It's hard to make good HTML sitemaps for large sites.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo
Consistent with docsD2-C820
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
“Better rely on hubs like category pages that link out to your important pages.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Used byrequirement DEV-URL-04
- Extends D1-C040 Day 1: URL discovery works through links: a homepage links to section pages, which link to further pages.
- Extends D1-C202 Day 1: Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they…
- Extended by D2-C838 Day 2: Answering whether HTML sitemaps still matter for large sites, Google said a site that wants one can make one…
Google's Q&A slide on HTML sitemaps added that category hub pages linking to a site's important pages have the extra benefit that they may also be useful to users.
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo
Used byrequirement DEV-URL-04
Answering whether HTML sitemaps still matter for large sites, Google said a site that wants one can make one, but that the effort is better spent on hub pages such as category pages.
“if you want to make one, knock yourself out, but I will focus on some better things like hub pages, category pages”
Speaker not identifiedEvidence transcript
Used byrequirement DEV-URL-04glossary term Hub pages
- Extends D2-C820 Day 2: Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their…
QQuestion and answers
An audience member asked whether a product detail page that is out of stock for two to three months should keep returning 200 with links to similar products, or be 302-redirected to a similar product or to its parent product listing page.
From the audienceEvidence slide photo, transcript
Not in docsD2-C010
Google answered on a Q&A slide that whether to keep a temporarily out-of-stock product page or redirect it depends on how important that product page is to users.
“It depends on the importance of the PDPs to the users.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Things
Used byrequirement DEV-ERR-04
- Extended by D2-C841 Day 2: Google contrasted out-of-stock products a buyer will wait for, such as a rare watch battery that matters to…
Consistent with docsD2-C011
Google's Q&A slide on out-of-stock product pages said users might wait months for some products, or even pre-order them if the site offers pre-ordering.
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Used byrequirement DEV-ERR-04
Google contrasted out-of-stock products a buyer will wait for, such as a rare watch battery that matters to them, with easily replaced products, such as a particular cheese, for which the buyer simply picks a similar product instead of waiting.
Speaker not identifiedEvidence transcript
- Extends D2-C010 Day 2: Google answered on a Q&A slide that whether to keep a temporarily out-of-stock product page or redirect it…
QQuestion and answers
An audience member asked whether the sitemap link should be included in robots.txt, noting that many websites include it but that it was missing from an earlier robots.txt slide by a Google speaker, which the question called 'Gary's slide'.
From the audienceEvidence slide photo, transcript
Things
Confirmed by docsD2-C017
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
“If it's included in the robots.txt file, any crawler can pick your sitemaps up”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Things
Used byrequirement DEV-URL-05
- Extends D1-C084 Day 1: Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as…
- Extends D1-C079 Day 1: The robots.txt session worked through an example file with three groups: a default group that disallows…
- Extended by D2-C843 Day 2: Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a…
Consistent with docsD2-C018
Google's Q&A slide said that a sitemap not listed in robots.txt has to be submitted instead, for example in Search Console.
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Used byrequirement DEV-URL-05
Google said listing the sitemap in robots.txt is fine, as many websites do.
“you can include it in robots.txt. No problem whatsoever.”
Speaker not identifiedEvidence transcript
Things
Used byrequirement DEV-URL-05
- Extends D1-C084 Day 1: Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as…
Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.
Speaker not identifiedEvidence transcript
Things
Used byrequirement DEV-URL-05
- Extends D2-C017 Day 2: Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler…
QQuestion and answers
An audience member noted that a robots.txt block does not mean noindex and that explaining this to developers is a constant struggle, and asked why Google does not treat a robots.txt block as a noindex.
From the audienceEvidence slide photo, transcript
Things
Not in docsD2-C020
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
“Simply put, it's because of sites that are extremely important and like to disallow their most important pages, either accidentally or out of ignorance.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Used byrequirement DEV-IDX-01
Confirmed by docsD2-C021
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
Speaker Gary Illyes, Cherry PrommawinEvidence slide photo, transcript
Things
Used byrequirement DEV-IDX-01
- Extends D1-C105 Day 1: URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
- Extended by D2-C846 Day 2: Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the…
- Repeated by D2-C851 Day 2: When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John…
- Extended by D2-C933 Day 2: Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt…
Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site, such as a national tax authority, that blocks very important PDF files with robots.txt: Google cannot index the PDFs' content but can at least show their URLs in search results.
Speaker not identifiedEvidence transcript
Used byrequirement DEV-IDX-01
- Extends D2-C020 Day 2: Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some…
Google said very few URLs disallowed by robots.txt are in its index, compared with the index as a whole (no figure given).
Speaker not identifiedEvidence transcript
Things
Used byrequirement DEV-IDX-01
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
“if a URL is important, then it might get indexed even if it's disallowed by robots.txt. So the URL gets indexed, not the content.”
Speaker not identifiedEvidence transcript
Things
Used byrequirement DEV-IDX-01
Said on stage 4
QQuestion and answer
StageD2-C839
An audience member asked how several sloppy migrations on the same domain can affect Googlebot's crawling.
From the audienceEvidence transcript
Things
Google answered that several sloppy migrations on one domain can cause many effects in the short term, and noted that migrations concern indexing as well as crawling, a subject a later Day 2 talk would cover.
Speaker not identifiedEvidence transcript
StageD2-C847
Opening Day 2, Google recapped that Day 1 began with how Search works and then covered crawling, including basics of how the internet works such as TCP/IP, so that attendees could understand how crawlers work.
Speaker not identifiedEvidence transcript
- Repeats D1-C317 Day 1: Google's crawling talk opened with how the internet works (TCP/IP, internet service providers, IP addresses…
Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.
Speaker not identifiedEvidence transcript
Things
- Repeats D1-C329 Day 1: Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as…
What Google's documentation says 7
Google's ecommerce documentation recommends linking from menus to category pages, from category pages to sub-category pages and from sub-category pages to all product pages; where not every product can be linked, it recommends a sitemap or a Merchant Center feed.
Publisher Google Search Central
Things
Used byrequirement DEV-URL-04
Google's ecommerce documentation says Google can infer a page's relative importance within a site from its internal links, such as how many links point to the page and how many links Google must follow to reach it.
Publisher Google Search Central
Used byrequirement DEV-URL-04
A 2005 Search Central blog post, now marked as possibly outdated, said Google encouraged HTML sitemaps because they help users navigate a site and a clear hierarchy of text links helps Google index it.
Publisher Search Central blog (26 September 2005)
Google's guide to temporarily pausing an online business recommends that a shop expecting to sell again within weeks or months stays online with limited functionality, such as a disabled cart, and updates its Product structured data to show current availability.
Publisher Google Search Central
Things
Used byrequirements DEV-ERR-04, DEV-SDA-08, DEV-SRV-03
Google's redirect documentation says that with a temporary redirect, such as a 302, Google Search shows the source page in search results and does not use the redirect as a signal that the target should be canonical.
Publisher Google Search Central
Things
Used byrequirements DEV-CAN-01, DEV-ERR-04
Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.
Publisher Google Search Central
Things
Used byrequirement DEV-IDX-01glossary term noindex
- Extends D1-C106 Day 1: The noindex rule consumes crawl budget, because Google must fetch the page to see it.
Google's sitemap guide says Google uses lastmod only when it is consistently and verifiably accurate, and counts a change to the main content, the structured data or the links of a page as significant, but not a changed copyright date.
Publisher Google Search Central
Used byrequirement DEV-URL-05
Analysis by the author 5
AnalysisD2-C004
Google's answer did not say whether HTML sitemaps pass link equity. On a large site, check that every important page is linked from a category or hub page in the normal navigation instead of relying on an HTML sitemap page to reach it.
Author Ibrahim Anjro
AnalysisD2-C008
Google's view of HTML sitemaps has moved: a 2005 blog post encouraged them, the current sitemap and ecommerce documentation does not mention them, and the 2026 Q&A slide pointed large sites to category hub pages instead.
Author Ibrahim Anjro
Things
AnalysisD2-C014
Keep a product page that is out of stock for a few months live with a 200 status, show its availability on the page and in Product structured data, and offer pre-ordering where possible when users would wait for the product. Redirect it only when users would rather switch to a similar product than wait.
Author Ibrahim Anjro
Used byrequirement DEV-ERR-04
AnalysisD2-C023
Explain robots.txt and noindex to developers as two separate controls: robots.txt controls crawling, noindex controls indexing. To keep a page out of Search, let Google crawl it and serve noindex; a robots.txt disallow alone can leave the bare URL in results.
Author Ibrahim Anjro
Used byrequirement DEV-IDX-01
AnalysisD2-C849
Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.
Author Ibrahim Anjro
Things
- Extends D1-C110 Day 1: A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google…
Links to other sessions 14
- D2-C017 Day 2 · Welcome to indexing day!
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
extendsD1-C079 Day 1 · How Google interprets robots.txtThe robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.
- D2-C017 Day 2 · Welcome to indexing day!
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
extendsDocs D1-C084 Day 1 · How Google interprets robots.txtGoogle supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
- D2-C020 Day 2 · Welcome to indexing day!
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
extendsAnalysis D1-C110 Day 1 · How Google thinks about crawl budgetA URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
- D2-C021 Day 2 · Welcome to indexing day!
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
extendsD1-C105 Day 1 · How Google thinks about crawl budgetURLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
- Docs D2-C022 Day 2 · Welcome to indexing day!
Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.
extendsD1-C106 Day 1 · How Google thinks about crawl budgetThe noindex rule consumes crawl budget, because Google must fetch the page to see it.
- D2-C820 Day 2 · Welcome to indexing day!
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
extendsD1-C040 Day 1 · How Search works and where's AI?URL discovery works through links: a homepage links to section pages, which link to further pages.
- D2-C820 Day 2 · Welcome to indexing day!
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
extendsStage D1-C202 Day 1 · How Search works and where's AI?Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they usually link out to new or updated pages.
- Stage D2-C842 Day 2 · Welcome to indexing day!
Google said listing the sitemap in robots.txt is fine, as many websites do.
extendsDocs D1-C084 Day 1 · How Google interprets robots.txtGoogle supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
- Stage D2-C846 Day 2 · Welcome to indexing day!
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
extendsAnalysis D1-C110 Day 1 · How Google thinks about crawl budgetA URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
- Analysis D2-C849 Day 2 · Welcome to indexing day!
Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.
extendsAnalysis D1-C110 Day 1 · How Google thinks about crawl budgetA URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
- Stage D2-C933 Day 2 · Using images to your advantage and Engaging Search users with videos
Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt is not shown by its URL, because Google sees no good reason to show a video URL alone.
extendsD2-C021 Day 2 · Welcome to indexing day!Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
- Stage D2-C847 Day 2 · Welcome to indexing day!
Opening Day 2, Google recapped that Day 1 began with how Search works and then covered crawling, including basics of how the internet works such as TCP/IP, so that attendees could understand how crawlers work.
repeatsStage D1-C317 Day 1 · How crawling worksGoogle's crawling talk opened with how the internet works (TCP/IP, internet service providers, IP addresses, URLs, DNS and HTTP), because crawling can only be understood with that wider scope.
- Stage D2-C848 Day 2 · Welcome to indexing day!
Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.
repeatsStage D1-C329 Day 1 · How crawling worksApart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.
- Stage D2-C851 Day 2 · Controlling indexing
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
repeatsD2-C021 Day 2 · Welcome to indexing day!Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.