Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Thing · Standard
Sitemaps
Files that list a site's URLs (with optional last-modified dates) so search engines can discover them.
- Claims
- 41
- In Google’s docs
- 7
- Said at the event
- 24
- Not in docs
- 3
- Kit items
- 17
Google’s documentation 7
Documented in
Google treats nofollow as a hint for crawling, not a block: nofollow links will generally not be followed, but the linked pages may still be crawled if Google finds them through sitemaps or other links. To stop Google fetching URLs on your own site, use a robots.txt disallow rule.
Google Search Central, Search Central blog (10 September 2019) · Day 1 · How Google thinks about crawl budget
Google's ecommerce documentation recommends linking from menus to category pages, from category pages to sub-category pages and from sub-category pages to all product pages; where not every product can be linked, it recommends a sitemap or a Merchant Center feed.
Google Search Central · Day 2 · Welcome to indexing day!
Google's sitemap guide says Google uses lastmod only when it is consistently and verifiably accurate, and counts a change to the main content, the structured data or the links of a page as significant, but not a changed copyright date.
Google Search Central · Day 2 · Welcome to indexing day!
Google's canonical guide ranks the ways to signal a preferred canonical by strength: redirects and rel=canonical annotations are strong signals, sitemap inclusion is a weak signal, and combining methods makes them more effective.
Google Search Central · Day 2 · Handling web duplication
Google's site move guide says to submit a Change of Address in Search Console when moving from one domain or subdomain to another, to submit the new sitemap, and to change internal links on the new site from the old URLs to the new ones.
Google Search Central · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Google's hreflang guide says its three methods (link elements in the HTML head, an HTTP Link header, which suits non-HTML files such as PDFs, and an XML sitemap) are equivalent; using several at once is allowed but brings no benefit in Search and is harder to manage.
Google Search Central · Day 2 · Focusing on Internationalisation and Localisation
Said at the event 24
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Label
Grade
Day
Not in docs 3
Video sitemaps are not critical but good to have, because Google ingests video sitemaps much more often than it can process HTML pages.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
Not in docsD3-C610
Google estimated that processing a sitemap takes about 24 hours on average, with a minimum of minutes.
Not in docsD3-C611
If a sitemap is useful to Google and the site is of high quality, Google tries to refetch the sitemap within 14 days at most.
Consistent with docs 11
An XML sitemap, a format from around 2005, gives Google, other search engines and potentially AI systems a list of the URLs a site wants crawled; Google called it nothing fancy but said it still uses sitemaps.
Gary Illyes · Day 1 · How crawling works
Asked how to plan a migration that does not leave many URLs unindexed, a Google panelist said the answer is probably not sitemaps: decide what matters from the business's perspective (for example whether to consolidate languages); listing the new URLs in a sitemap is probably a good idea and cannot hurt, but it is not the main tool.
Consistent with docsD2-C018
Google's Q&A slide said that a sitemap not listed in robots.txt has to be submitted instead, for example in Search Console.
Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.
Consistent with docsD2-C387
Google named three considerations when picking a representative URL: hijacking across pages or sites, user experience (the page can load, meta refresh, security) and site-owner signals (redirects, rel=canonical, sitemaps).
Consistent with docsD2-C390
Clear site-owner signals about which URL should be canonical make a big difference to Google's choice; the speaker named redirects, listing only the preferred URL in sitemaps, and rel=canonical, which the speaker said also helps a bit.
Before a migration launches, a community speaker advised freezing the inventory (CMS export, crawl, sitemap, search data and logs), saving all content and approving the redirect map.
David Carrasco Pamies · Day 2 · Lightning session E: Managing Duplicates and Site Moves
One of the most reliable ways to make sure Google finds an image is to include it in an img element in the HTML; Gary Illyes named image sitemaps as the other method.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
Consistent with docsD2-C552
Google's slide listed seven key factors for video SEO success: high-quality video content, a dedicated watch page, compelling titles and descriptions, relevant thumbnails, video markup, fast-loading pages and sitemap inclusion.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
A video sitemap or video feed tells Google which URLs carry videos so it can visit those URLs to double-check; without one, Google has to check every page individually.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
Consistent with docsD3-C612
Google may never fetch a lower-quality site's sitemap again: once it figures out the site is of lower quality, it no longer wants to fetch the sitemap.
Confirmed by docs 8
Google finds what to crawl mainly by extracting URLs from previously crawled pages, and additionally from sitemaps.
Gary Illyes · Day 1 · How crawling works
There is no sweet spot to find for a sitemap: technically it should list every URL you want indexed, so that Google can find each of them.
Confirmed by docsD2-C017
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
Google said listing the sitemap in robots.txt is fine, as many websites do.
The technical steps of a community speaker's domain consolidation included submitting new sitemaps, filing a change of address in Search Console and updating internal links so the new pages did not rely on redirects alone.
Martyna Ağanoğlu · Day 2 · Lightning session E: Managing Duplicates and Site Moves
Besides img elements, image sitemaps tell Google about images: they are XML sitemaps that list, under a page's loc entry, the locations of the images on that page.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
Confirmed by docsD2-C556
Including videos in a sitemap helps Google discover all of a site's video content.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
When the HTML head is not suitable for a page, hreflang can be given by other methods instead, such as listing the language versions in a sitemap.
Nothing to verify 2
StageD1-C474
An audience member asked where the sweet spot is for a sitemap that misses nothing but does not overflow Search Console.
From the audience · Day 1 · Q&A
An audience member asked whether the sitemap link should be included in robots.txt, noting that many websites include it but that it was missing from an earlier robots.txt slide by a Google speaker, which the question called 'Gary's slide'.
From the audience · Day 2 · Welcome to indexing day!
Press and analysis 10
AnalysisD1-C067
A page with no internal links depends on sitemaps alone to be found, so it is discovered slowly and attracts little crawl demand.
Ibrahim Anjro · Day 1 · How crawling works
AnalysisD2-C008
Google's view of HTML sitemaps has moved: a 2005 blog post encouraged them, the current sitemap and ecommerce documentation does not mention them, and the 2026 Q&A slide pointed large sites to category hub pages instead.
Ibrahim Anjro · Day 2 · Welcome to indexing day!
AnalysisD2-C189
Build market and language selectors as plain <a href> links to each alternate URL, not buttons or script handlers; otherwise the language versions have no internal links and depend on sitemaps to be found, which is slow.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
AnalysisD2-C392
Google's duplication talk described rel=canonical as something that 'also helps a bit', while Google's canonical guide calls it a strong signal alongside redirects and calls sitemap inclusion weak; treat redirects and rel=canonical as the main levers and sitemaps as support.
Ibrahim Anjro · Day 2 · Handling web duplication
AnalysisD2-C400
Before a migration or template change, align every canonical signal for the preferred URL (redirects, rel=canonical, internal links, sitemap entries, hreflang and working HTTPS); Google says it follows the site owner only when the signals agree.
Ibrahim Anjro · Day 2 · Handling web duplication
AnalysisD2-C431
A leader reachable only through a canonical and a leader with weak link equity share one fix: point internal links, sitemap entries and redirects at the URL chosen as canonical, so the leader is both reachable for users and the strongest page in its group.
Ibrahim Anjro · Day 2 · Lightning session E: Managing Duplicates and Site Moves
AnalysisD2-C530
Hiding an image as a CSS background is a fragile way to keep it out of Google: it only stops extraction from that page, so the same image URL used in an img element elsewhere or listed in a sitemap can still be indexed; the documented robots.txt or noindex X-Robots-Tag methods are the reliable route.
Ibrahim Anjro · Day 2 · Using images to your advantage and Engaging Search users with videos
AnalysisD2-C557
Give each important video its own watch page where the video is the main content, add VideoObject markup and list the video in a video sitemap; a video buried in a long article lacks the dedicated watch page that Google listed as a key factor.
Ibrahim Anjro · Day 2 · Using images to your advantage and Engaging Search users with videos
AnalysisD2-C574
Audit hreflang per cluster rather than per page: check that every page lists itself and all its alternates, that every alternate links back, and that only one method (HTML head, HTTP header or sitemap) supplies the annotations.
Ibrahim Anjro · Day 2 · Focusing on Internationalisation and Localisation
AnalysisD3-C638
By Google's own averages, getting found and crawled takes far longer than indexing (about 20 hours to discover a URL and 30 days to refresh one, against 1.5 hours to index), so for faster results work on discovery: internal links from often-crawled pages and accurate sitemaps.
Ibrahim Anjro · Day 3 · How long does it take to..?
Built on these claims 17
Kit items about Sitemaps: their own words name it, or several of the claims they rest on do.
Developer requirements 13
Publish XML sitemaps listing only canonical, indexable URLs with accurate lastmod
Rests on 11 claims, 11 of them naming Sitemaps; its own words name Sitemaps
Make all canonical signals agree: redirects, rel=canonical, internal links, sitemaps and hreflang
Rests on 6 claims, 4 of them naming Sitemaps; its own words name Sitemaps
Rests on 14 claims, 3 of them naming Sitemaps; its own words name Sitemaps
List videos in a video sitemap
Rests on 3 claims, 3 of them naming Sitemaps; its own words name Sitemaps
List important images in an image sitemap
Rests on 2 claims, 2 of them naming Sitemaps; its own words name Sitemaps
8 more
Supply hreflang through one method only: HTML head, HTTP Link header or sitemap
Rests on 3 claims, 2 of them naming Sitemaps; its own words name Sitemaps
Link every indexable page from the normal navigation or a hub page
Rests on 8 claims, 2 of them naming Sitemaps; its own words name Sitemaps
Give each important video its own watch page
Rests on 6 claims, 2 of them naming Sitemaps; its own words name Sitemaps
Never leave staging, preview or mirror copies of the site publicly crawlable
Rests on 2 claims, 1 of them naming Sitemaps
Do not use noindex, nofollow or crawl-delay to save crawl budget
Rests on 6 claims, 1 of them naming Sitemaps; its own words name Sitemaps
Keep images out of Search with robots.txt or X-Robots-Tag, not with CSS tricks
Rests on 5 claims, 1 of them naming Sitemaps; its own words name Sitemaps
Build language and market selectors as <a href> links to each alternate URL
Rests on 2 claims, 1 of them naming Sitemaps; its own words name Sitemaps
Rests on 20 claims, 1 of them naming Sitemaps; its own words name Sitemaps
Also inglossary terms Video sitemap, Change of Address tool, nofollow, Hub pages
Connected things 16
Most often named with it
Things named in the same claim, with the number of claims they share.
- Redirects 8
- robots.txt 8
- Canonicalization 5
- rel=canonical 5
- hreflang 4
- Search Console 4
- Disallow 2
- Main content 2
- Allow 1
- Crawl demand 1
- Merchant Center 1
- nofollow 1
- noindex 1
- Site move 1
- Structured data 1
- X-Robots-Tag 1
Topics that feature it
- Sitemaps 28
- URL discovery 7
- Redirects, site moves and alternate names 6
- Canonical selection 5
- Video in Search 5
- robots.txt rules 5
- Directives and crawl budget 3
- How long Google's processes take 3
- Images 3
- hreflang annotations 3
- Crawlable links 2
- What's new in Search Console 2
- Canonical graphs: chains, loops and leaders 1
- Crawl demand 1
- Implementing structured data 1
- Measuring success in the AI era 1
- Page language and language versions 1