Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
A sitemap line in robots.txt lets any crawler find the sitemap; a sitemap not listed there has to be submitted, for example in Search Console. For HTML sitemaps on large sites, a Google Q&A slide said good HTML sitemaps are hard to build at that scale (said at the event) and advised relying instead on category hub pages that link to the important pages; Google's ecommerce guide recommends the same linking, with a sitemap or Merchant Center feed where not every product can be linked. As a canonical signal, listing only the preferred URL in sitemaps was named on stage among the signals that make a big difference, but Google's canonical guide rates sitemap inclusion as weak next to redirects and rel=canonical. Image and video sitemaps help Google find media, and an XML sitemap is one of three equivalent ways to declare hreflang. Author’s view: sitemaps support discovery but do not replace internal links, because pages known only from sitemaps are found slowly. Day 3 timed sitemaps and tied them to quality: Google estimated that processing a sitemap takes about 24 hours on average and that it tries to refetch a useful sitemap of a high-quality site within 14 days at most (neither figure is in Google's docs), while it may never fetch a lower-quality site's sitemap again. The second recordings added Google's framing: Gary Illyes said Google finds what to crawl mainly by extracting URLs from crawled pages and additionally from sitemaps, an XML format from around 2005 that is 'nothing fancy' but still used, and in the Day 1 Q&A Google said there is no sweet spot to find for a sitemap: technically it should list every URL you want indexed, so that Google can find each of them; for a migration, listing the new URLs in a sitemap cannot hurt but is not the main tool. Day 2's opening Q&A said a site that wants an HTML sitemap can make one but the effort is better spent on hub pages, and that listing a sitemap in robots.txt is fine but makes it public. For migrations, Google's site move guide and a community case study both submit the new sitemap. Gary Illyes said video sitemaps are not critical but good to have, because Google ingests them much more often than it can process HTML pages and they tell it which URLs carry videos (said at the event).
Things in this topic 12
Counts are claims that name the thing. All things
What to do
Google finds what to crawl mainly by extracting URLs from previously crawled pages, and additionally from sitemaps.
Speaker Gary IllyesIn Day 1, 14:05 · How crawling worksEvidence transcript
Used byrequirement DEV-URL-05
An XML sitemap, a format from around 2005, gives Google, other search engines and potentially AI systems a list of the URLs a site wants crawled; Google called it nothing fancy but said it still uses sitemaps.
Speaker Gary IllyesIn Day 1, 14:05 · How crawling worksEvidence transcript
Used byrequirement DEV-URL-05
An audience member asked where the sweet spot is for a sitemap that misses nothing but does not overflow Search Console.
From the audienceIn Day 1, 16:35 · Q&AEvidence transcript
There is no sweet spot to find for a sitemap: technically it should list every URL you want indexed, so that Google can find each of them.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-05
Asked how to plan a migration that does not leave many URLs unindexed, a Google panelist said the answer is probably not sitemaps: decide what matters from the business's perspective (for example whether to consolidate languages); listing the new URLs in a sitemap is probably a good idea and cannot hurt, but it is not the main tool.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-CAN-09
An audience member asked whether HTML sitemaps still matter for large sites, whether they improve Googlebot's crawl discovery and whether they pass link equity to deeper pages.
From the audienceIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo
Google answered on a Q&A slide that good HTML sitemaps are hard to make for large sites.
“It's hard to make good HTML sitemaps for large sites.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
“Better rely on hubs like category pages that link out to your important pages.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Used byrequirement DEV-URL-04
An audience member asked whether the sitemap link should be included in robots.txt, noting that many websites include it but that it was missing from an earlier robots.txt slide by a Google speaker, which the question called 'Gary's slide'.
From the audienceIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
“If it's included in the robots.txt file, any crawler can pick your sitemaps up”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Used byrequirement DEV-URL-05
Google's Q&A slide said that a sitemap not listed in robots.txt has to be submitted instead, for example in Search Console.
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Used byrequirement DEV-URL-05
Clear site-owner signals about which URL should be canonical make a big difference to Google's choice; the speaker named redirects, listing only the preferred URL in sitemaps, and rel=canonical, which the speaker said also helps a bit.
Speaker John MuellerIn Day 2, 11:55 · Handling web duplicationEvidence slide photo, transcript
Used byrequirements DEV-CAN-05, DEV-URL-05
Including videos in a sitemap helps Google discover all of a site's video content.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence slide photo, transcript
Used byrequirement DEV-VID-04glossary term Video sitemap
Answering whether HTML sitemaps still matter for large sites, Google said a site that wants one can make one, but that the effort is better spent on hub pages such as category pages.
“if you want to make one, knock yourself out, but I will focus on some better things like hub pages, category pages”
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
Used byrequirement DEV-URL-04glossary term Hub pages
Google said listing the sitemap in robots.txt is fine, as many websites do.
“you can include it in robots.txt. No problem whatsoever.”
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
Used byrequirement DEV-URL-05
Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
Used byrequirement DEV-URL-05
The technical steps of a community speaker's domain consolidation included submitting new sitemaps, filing a change of address in Search Console and updating internal links so the new pages did not rely on redirects alone.
Speaker Martyna AğanoğluIn Day 2, 12:05 · Lightning session E: Managing Duplicates and Site MovesEvidence transcript
Used byrequirement DEV-CAN-09glossary term Change of Address tool
Besides img elements, image sitemaps tell Google about images: they are XML sitemaps that list, under a page's loc entry, the locations of the images on that page.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirement DEV-IMG-04
Video sitemaps are not critical but good to have, because Google ingests video sitemaps much more often than it can process HTML pages.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirement DEV-VID-04glossary term Video sitemap
A video sitemap or video feed tells Google which URLs carry videos so it can visit those URLs to double-check; without one, Google has to check every page individually.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirement DEV-VID-04glossary term Video sitemap
When the HTML head is not suitable for a page, hreflang can be given by other methods instead, such as listing the language versions in a sitemap.
Speaker GoogleIn Day 2, 14:15 · Focusing on Internationalisation and LocalisationEvidence transcript
Used byrequirement DEV-INT-05
Google's ecommerce documentation recommends linking from menus to category pages, from category pages to sub-category pages and from sub-category pages to all product pages; where not every product can be linked, it recommends a sitemap or a Merchant Center feed.
Publisher Google Search CentralAnnotates Day 2, 10:15 · Welcome to indexing day!
Used byrequirement DEV-URL-04
A 2005 Search Central blog post, now marked as possibly outdated, said Google encouraged HTML sitemaps because they help users navigate a site and a clear hierarchy of text links helps Google index it.
Publisher Search Central blog (26 September 2005)Annotates Day 2, 10:15 · Welcome to indexing day!
Google's sitemap guide says Google uses lastmod only when it is consistently and verifiably accurate, and counts a change to the main content, the structured data or the links of a page as significant, but not a changed copyright date.
Publisher Google Search CentralAnnotates Day 2, 10:15 · Welcome to indexing day!
Used byrequirement DEV-URL-05
Google's canonical guide ranks the ways to signal a preferred canonical by strength: redirects and rel=canonical annotations are strong signals, sitemap inclusion is a weak signal, and combining methods makes them more effective.
Publisher Google Search CentralAnnotates Day 2, 11:55 · Handling web duplication
Used byrequirements DEV-CAN-05, DEV-URL-05glossary term rel=canonical
Google's site move guide says to submit a Change of Address in Search Console when moving from one domain or subdomain to another, to submit the new sitemap, and to change internal links on the new site from the old URLs to the new ones.
Publisher Google Search CentralAnnotates Day 2, 12:05 · Lightning session E: Managing Duplicates and Site Moves
Used byrequirement DEV-CAN-09glossary term Change of Address tool
Google's hreflang guide says its three methods (link elements in the HTML head, an HTTP Link header, which suits non-HTML files such as PDFs, and an XML sitemap) are equivalent; using several at once is allowed but brings no benefit in Search and is harder to manage.
Publisher Google Search CentralAnnotates Day 2, 14:15 · Focusing on Internationalisation and Localisation
Used byrequirement DEV-INT-05
Google's answer did not say whether HTML sitemaps pass link equity. On a large site, check that every important page is linked from a category or hub page in the normal navigation instead of relying on an HTML sitemap page to reach it.
Author Ibrahim AnjroAnnotates Day 2, 10:15 · Welcome to indexing day!
Google's view of HTML sitemaps has moved: a 2005 blog post encouraged them, the current sitemap and ecommerce documentation does not mention them, and the 2026 Q&A slide pointed large sites to category hub pages instead.
Author Ibrahim AnjroAnnotates Day 2, 10:15 · Welcome to indexing day!
Google's duplication talk described rel=canonical as something that 'also helps a bit', while Google's canonical guide calls it a strong signal alongside redirects and calls sitemap inclusion weak; treat redirects and rel=canonical as the main levers and sitemaps as support.
Author Ibrahim AnjroAnnotates Day 2, 11:55 · Handling web duplication
Give each important video its own watch page where the video is the main content, add VideoObject markup and list the video in a video sitemap; a video buried in a long article lacks the dedicated watch page that Google listed as a key factor.
Author Ibrahim AnjroAnnotates Day 2, 13:50 · Using images to your advantage and Engaging Search users with videos
Used byrequirement DEV-VID-02
Google estimated that processing a sitemap takes about 24 hours on average, with a minimum of minutes.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence slide photo, transcript
If a sitemap is useful to Google and the site is of high quality, Google tries to refetch the sitemap within 14 days at most.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence slide photo, transcript
Google may never fetch a lower-quality site's sitemap again: once it figures out the site is of lower quality, it no longer wants to fetch the sitemap.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence slide photo, transcript
Google finds what to crawl mainly by extracting URLs from previously crawled pages, and additionally from sitemaps.
URL discovery works through links: a homepage links to section pages, which link to further pages.
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
URL discovery works through links: a homepage links to section pages, which link to further pages.
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they usually link out to new or updated pages.
Answering whether HTML sitemaps still matter for large sites, Google said a site that wants one can make one, but that the effort is better spent on hub pages such as category pages.
Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.
Google said listing the sitemap in robots.txt is fine, as many websites do.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
The technical steps of a community speaker's domain consolidation included submitting new sitemaps, filing a change of address in Search Console and updating internal links so the new pages did not rely on redirects alone.
Asked how to plan a migration that does not leave many URLs unindexed, a Google panelist said the answer is probably not sitemaps: decide what matters from the business's perspective (for example whether to consolidate languages); listing the new URLs in a sitemap is probably a good idea and cannot hurt, but it is not the main tool.
Google estimated that processing a sitemap takes about 24 hours on average, with a minimum of minutes.
An XML sitemap, a format from around 2005, gives Google, other search engines and potentially AI systems a list of the URLs a site wants crawled; Google called it nothing fancy but said it still uses sitemaps.
If a sitemap is useful to Google and the site is of high quality, Google tries to refetch the sitemap within 14 days at most.
An XML sitemap, a format from around 2005, gives Google, other search engines and potentially AI systems a list of the URLs a site wants crawled; Google called it nothing fancy but said it still uses sitemaps.
Google may never fetch a lower-quality site's sitemap again: once it figures out the site is of lower quality, it no longer wants to fetch the sitemap.
Google's crawl scheduler very likely deprioritises a URL when the URL or its site is known to be historically spammy.
Publish XML sitemaps listing only canonical, indexable URLs with accurate lastmod
Rests on 11 claims, 10 of them in this topic
Rests on 14 claims, 3 of them in this topic
Link every indexable page from the normal navigation or a hub page
Rests on 8 claims, 3 of them in this topic
List videos in a video sitemap
Rests on 3 claims, 3 of them in this topic
Make all canonical signals agree: redirects, rel=canonical, internal links, sitemaps and hreflang
Rests on 6 claims, 2 of them in this topic
Supply hreflang through one method only: HTML head, HTTP Link header or sitemap
Rests on 3 claims, 2 of them in this topic
List important images in an image sitemap
Rests on 2 claims, 1 of them in this topic
Give each important video its own watch page
Rests on 6 claims, 1 of them in this topic