robots.txt rules

Day 1: Crawling 52

Shown on screen 3

Said on stage 38

What Google's documentation says 8

Analysis by the author 3

Day 2: Indexing 37

Shown on screen 10

Said on stage 16

What Google's documentation says 4

Analysis by the author 7

Day 3: Serving: Ranking, Search Console, and Performance 7

Shown on screen 1

Said on stage 2

What Google's documentation says 2

Analysis by the author 2

Across days and sessions 50

1. Analysis D1-C433 Day 1 · Q&A

   The crawler best practices mentioned on stage match the public IETF Internet-Draft 'Crawler best practices' (draft-illyes-aipref-cbcp-00, July 2025), which says self-declared research crawlers, including privacy and malware discovery crawlers, may exempt themselves from any of its practices with a rationale; it is a draft, not Google documentation.

   extends
   Stage D1-C432 Day 1 · Q&A

   A Google panelist said they were working on a set of crawler best practices and offering research, malware-scanning and privacy crawlers an exemption from following them, apparently because such crawlers sometimes need to ignore robots.txt or probe URLs that other crawlers would not touch (the reason is a best reading: the audio has 'don't need', which would not explain an exemption).

2. Analysis D1-C470 Day 1 · Q&A

   Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.

   extends
   Stage D1-C469 Day 1 · Q&A

   Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.

3. Stage D1-C512 Day 1 · How Google interprets robots.txt

   Robots.txt is now an IETF standard, RFC 9309, supported by basically all major crawlers; Google said it follows it because it is the right thing to do: anyone who wants to opt out of crawling should be able to.

   extends
   Stage D1-C329 Day 1 · How crawling works

   Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.

4. Stage D1-C513 Day 1 · How Google interprets robots.txt

   The standard's official name is the Robots Exclusion Protocol (REP). Google stressed that it was designed in 1994 only to control which automated clients may access what on a site, and has nothing to do with how the content is used.

   extends
   Docs D1-C086 Day 1 · How Google interprets robots.txt

   Google-Extended is a control token, not a crawler with its own user agent string. It decides whether crawled content may be used to train future Gemini models and to ground Gemini apps and Vertex AI.

5. Stage D1-C516 Day 1 · How Google interprets robots.txt

   Robots.txt was made extremely simple by design so that anyone could implement and understand it; it was extended as websites grew more complex, but it still has only three rules: user-agent (a named crawler, or * for every crawler), disallow and allow.

   extends
   Docs D1-C084 Day 1 · How Google interprets robots.txt

   Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.

6. Stage D1-C519 Day 1 · How Google interprets robots.txt

   A user-agent line with its allow and disallow rules forms a user-agent group, and one group can name several crawlers (the talk's example: Googlebot and Bingbot); give named crawlers their own group only when they need rules the * group should not grant to every crawler.

   extends
   Docs D1-C080 Day 1 · How Google interprets robots.txt

   Rules can be grouped for several crawlers by repeating user-agent lines. A crawler follows only the most specific group that names it.

7. Stage D1-C520 Day 1 · How Google interprets robots.txt

   In the talk's worked example, disallow /*/live/ blocks /science/live/ and /sports/live/, because * matches any number of characters, and an allow /science/live/ rule re-opens that one path.

   extends
   Slide D1-C079 Day 1 · How Google interprets robots.txt

   The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.

8. Stage D1-C520 Day 1 · How Google interprets robots.txt

   In the talk's worked example, disallow /*/live/ blocks /science/live/ and /sports/live/, because * matches any number of characters, and an allow /science/live/ rule re-opens that one path.

   extends
   Docs D1-C081 Day 1 · How Google interprets robots.txt

   When matching rules to a URL, Google uses the most specific rule by path length. If rules conflict, it uses the least restrictive one.

9. Stage D1-C521 Day 1 · How Google interprets robots.txt

   To let unnamed crawlers fetch only the homepage, use user-agent: *, disallow: / and allow: /$; the $ ends the match, so /$ means only the root path and /cats$ means exactly /cats.

   extends
   Slide D1-C079 Day 1 · How Google interprets robots.txt

   The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.

10. Stage D1-C521 Day 1 · How Google interprets robots.txt

    To let unnamed crawlers fetch only the homepage, use user-agent: *, disallow: / and allow: /$; the $ ends the match, so /$ means only the root path and /cats$ means exactly /cats.

    extends
    Docs D1-C082 Day 1 · How Google interprets robots.txt

    In robots.txt, * matches zero or more of any character and $ marks the end of the URL.

11. Stage D1-C527 Day 1 · How Google interprets robots.txt

    The talk's fix for giving one crawler both an extra rule and the rules of another group: simply add another group for that crawler. Google's spec combines all groups that name the same user agent into one.

    extends
    Docs D1-C080 Day 1 · How Google interprets robots.txt

    Rules can be grouped for several crawlers by repeating user-agent lines. A crawler follows only the most specific group that names it.

12. Stage D1-C528 Day 1 · How Google interprets robots.txt

    Search Console's robots.txt report shows the robots.txt file as Google last fetched it, with a version history and the errors and successes of each fetch.

    extends
    Docs D1-C140 Day 1 · How Google interprets robots.txt

    Search Console's robots.txt report shows the robots.txt files Google found for the top 20 hosts of a Domain property, with their fetch status, warnings and errors; to test whether a specific URL is blocked, Google's help page points to the URL Inspection tool and to Google's open-source robots.txt library.

13. Stage D1-C530 Day 1 · How Google interprets robots.txt

    Google said the robots.txt report uses the parser Google open-sourced at github.com/google/robotstxt.

    extends
    Docs D1-C140 Day 1 · How Google interprets robots.txt

    Search Console's robots.txt report shows the robots.txt files Google found for the top 20 hosts of a Domain property, with their fetch status, warnings and errors; to test whether a specific URL is blocked, Google's help page points to the URL Inspection tool and to Google's open-source robots.txt library.

14. Stage D1-C531 Day 1 · How Google interprets robots.txt

    In the talk's quiz, the answer for letting Googlebot crawl /staging/preview/ while keeping the rest of /staging/ closed to it was a user-agent: googlebot group with both rules, disallow /staging/ and allow /staging/preview/, rather than an allow in the * group or a Googlebot group with only the allow.

    extends
    Docs D1-C080 Day 1 · How Google interprets robots.txt

    Rules can be grouped for several crawlers by repeating user-agent lines. A crawler follows only the most specific group that names it.

15. Analysis D1-C532 Day 1 · How Google interprets robots.txt

    A typo in a rule name is not always ignored by Google: its open-source robots.txt parser deliberately accepts common misspellings of disallow (such as dissallow, dissalow and disalow) and of user-agent (useragent, user agent), but not of allow. Google's spec page does not mention typos, and other crawlers may be stricter, so spell rule names correctly.

    extends
    Stage D1-C525 Day 1 · How Google interprets robots.txt

    Google called robots.txt extremely forgiving: a typo in a path only blocks the wrong path, a typo in a rule name such as disallow makes Google ignore that line, and the rest of the file is still used.

16. Stage D1-C533 Day 1 · Lightning session B: Robots.txt

    Dave Smart said robots.txt groups are not additive: if a group matches a crawler's user agent, only that group's rules apply, so a googlebot group that blocks /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks.

    extends
    Docs D1-C080 Day 1 · How Google interprets robots.txt

    Rules can be grouped for several crawlers by repeating user-agent lines. A crawler follows only the most specific group that names it.

17. Slide D2-C017 Day 2 · Welcome to indexing day!

    Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.

    extends
    Slide D1-C079 Day 1 · How Google interprets robots.txt

    The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.

18. Slide D2-C017 Day 2 · Welcome to indexing day!

    Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.

    extends
    Docs D1-C084 Day 1 · How Google interprets robots.txt

    Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.

19. Slide D2-C020 Day 2 · Welcome to indexing day!

    Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.

    extends
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

20. Slide D2-C021 Day 2 · Welcome to indexing day!

    Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.

    extends
    Slide D1-C105 Day 1 · How Google thinks about crawl budget

    URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.

21. Docs D2-C022 Day 2 · Welcome to indexing day!

    Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.

    extends
    Slide D1-C106 Day 1 · How Google thinks about crawl budget

    The noindex rule consumes crawl budget, because Google must fetch the page to see it.

22. Slide D2-C024 Day 2 · How is HTML interpreted

    A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.

    extends
    Slide D1-C064 Day 1 · How crawling works

    The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

23. Slide D2-C025 Day 2 · How is HTML interpreted

    In Google's example fetch record, the robots policies that apply to a fetch, shown as the value opt-out-google-extended, travel with the fetched page into processing.

    extends
    Slide D1-C064 Day 1 · How crawling works

    The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

24. Analysis D2-C067 Day 2 · Controlling indexing

    nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.

    extends
    Stage D1-C482 Day 1 · Q&A

    Nearly all mainstream AI crawlers and AI systems use their own user agents, so robots.txt can set a separate policy for each; at the event OpenAI was said to have three or four, Anthropic a few and Microsoft some.

25. Docs D2-C069 Day 2 · Controlling indexing

    Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.

    extends
    Slide D1-C106 Day 1 · How Google thinks about crawl budget

    The noindex rule consumes crawl budget, because Google must fetch the page to see it.

26. Docs D2-C122 Day 2 · Lightning session D: Rendering and JavaScript

    Google says user-triggered fetchers, which fetch a URL because a user asked for it in a Google product (for example Gemini Notebook fetching URLs users add as sources, or Google-Agent acting on a user's request), generally ignore robots.txt rules.

    extends
    Stage D1-C434 Day 1 · Q&A

    User-initiated fetchers, such as a translation service fetching a page a user asked to translate, are a different category from crawlers and generally do not check robots.txt, because a user asked for the fetch.

27. Slide D2-C204 Day 2 · Lightning session D: Rendering and JavaScript

    Blocking JavaScript or API resources in robots.txt is a classic rendering mistake: if Google's renderer cannot fetch a resource, it cannot run it.

    extends
    Slide D1-C064 Day 1 · How crawling works

    The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

28. Analysis D2-C208 Day 2 · Lightning session D: Rendering and JavaScript

    A robots.txt carve-out works because Google applies the most specific matching rule, so Allow: /api/products/ beats Disallow: /api/ for product endpoints only; re-test a rendered page after every robots.txt change to script or API paths.

    extends
    Docs D1-C081 Day 1 · How Google interprets robots.txt

    When matching rules to a URL, Google uses the most specific rule by path length. If rules conflict, it uses the least restrictive one.

29. Analysis D2-C210 Day 2 · Lightning session D: Rendering and JavaScript

    An API or CDN on its own host needs its own robots.txt check: a blanket Disallow there, or a robots.txt that returns 5xx errors, can stop Google fetching the data a page renders from.

    extends
    Docs D1-C074 Day 1 · How crawling errors affect Search

    If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the cached copy for up to 30 days.

30. Stage D2-C296 Day 2 · What is Google friendly JavaScript

    If robots.txt blocks JavaScript that client-side code needs to render, Google cannot render the content that the JavaScript would produce.

    extends
    Slide D1-C064 Day 1 · How crawling works

    The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

31. Stage D2-C842 Day 2 · Welcome to indexing day!

    Google said listing the sitemap in robots.txt is fine, as many websites do.

    extends
    Docs D1-C084 Day 1 · How Google interprets robots.txt

    Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.

32. Stage D2-C843 Day 2 · Welcome to indexing day!

    Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.

    extends
    Slide D2-C017 Day 2 · Welcome to indexing day!

    Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.

33. Stage D2-C844 Day 2 · Welcome to indexing day!

    Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site, such as a national tax authority, that blocks very important PDF files with robots.txt: Google cannot index the PDFs' content but can at least show their URLs in search results.

    extends
    Slide D2-C020 Day 2 · Welcome to indexing day!

    Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.

34. Stage D2-C846 Day 2 · Welcome to indexing day!

    Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.

    extends
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

35. Stage D2-C846 Day 2 · Welcome to indexing day!

    Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.

    extends
    Slide D2-C021 Day 2 · Welcome to indexing day!

    Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.

36. Analysis D2-C849 Day 2 · Welcome to indexing day!

    Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.

    extends
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

37. Stage D2-C933 Day 2 · Using images to your advantage and Engaging Search users with videos

    Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt is not shown by its URL, because Google sees no good reason to show a video URL alone.

    extends
    Slide D2-C021 Day 2 · Welcome to indexing day!

    Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.

38. Slide D3-C613 Day 3 · How long does it take to..?

    Google estimated that a robots.txt update is picked up in about 24 hours, with a minimum of seconds and an end point of 25 hours on the slide.

    extends
    Docs D1-C085 Day 1 · How Google interprets robots.txt

    Google generally caches robots.txt for up to 24 hours and reads only the first 500 KiB of the file.

39. Stage D3-C614 Day 3 · How long does it take to..?

    Google said its service level objective is to refresh robots.txt every 24 hours, as RFC 9309 calls for, though delays happen now and then.

    extends
    Docs D1-C085 Day 1 · How Google interprets robots.txt

    Google generally caches robots.txt for up to 24 hours and reads only the first 500 KiB of the file.

40. Stage D3-C615 Day 3 · How long does it take to..?

    A site owner can submit robots.txt in Search Console to force Google to refresh it sooner.

    extends
    Docs D1-C140 Day 1 · How Google interprets robots.txt

    Search Console's robots.txt report shows the robots.txt files Google found for the top 20 hosts of a Domain property, with their fetch status, warnings and errors; to test whether a specific URL is blocked, Google's help page points to the URL Inspection tool and to Google's open-source robots.txt library.

41. Docs D3-C667 Day 3 · How long does it take to..?

    Google's robots.txt guide says its crawlers update their cached copy of a site's robots.txt every 24 hours, and that the Request a recrawl option in Search Console's robots.txt report refreshes it faster.

    extends
    Stage D1-C528 Day 1 · How Google interprets robots.txt

    Search Console's robots.txt report shows the robots.txt file as Google last fetched it, with a version history and the errors and successes of each fetch.

42. Stage D1-C434 Day 1 · Q&A

    User-initiated fetchers, such as a translation service fetching a page a user asked to translate, are a different category from crawlers and generally do not check robots.txt, because a user asked for the fetch.

    repeats
    Stage D1-C273 Day 1 · Lightning session A: Automation and AI

    A community speaker pointed out that, according to Google's documentation, user-triggered fetchers, which fetch a page because a user asked for it, generally ignore robots.txt rules.

43. Stage D2-C050 Day 2 · Controlling indexing

    John Mueller noted that robots.txt is defined by an internet standard, RFC 9309, which Google's Gary Illyes co-authored.

    repeats
    Stage D1-C512 Day 1 · How Google interprets robots.txt

    Robots.txt is now an IETF standard, RFC 9309, supported by basically all major crawlers; Google said it follows it because it is the right thing to do: anyone who wants to opt out of crawling should be able to.

44. Docs D2-C122 Day 2 · Lightning session D: Rendering and JavaScript

    Google says user-triggered fetchers, which fetch a URL because a user asked for it in a Google product (for example Gemini Notebook fetching URLs users add as sources, or Google-Agent acting on a user's request), generally ignore robots.txt rules.

    repeats
    Stage D1-C273 Day 1 · Lightning session A: Automation and AI

    A community speaker pointed out that, according to Google's documentation, user-triggered fetchers, which fetch a page because a user asked for it, generally ignore robots.txt rules.

45. Slide D2-C265 Day 2 · What is Google friendly JavaScript

    Blocked resources, one of Google's four common JavaScript indexing problems, means robots.txt disallowing the crawling of critical JavaScript files or API endpoints.

    repeats
    Slide D2-C204 Day 2 · Lightning session D: Rendering and JavaScript

    Blocking JavaScript or API resources in robots.txt is a classic rendering mistake: if Google's renderer cannot fetch a resource, it cannot run it.

46. Stage D2-C296 Day 2 · What is Google friendly JavaScript

    If robots.txt blocks JavaScript that client-side code needs to render, Google cannot render the content that the JavaScript would produce.

    repeats
    Slide D2-C204 Day 2 · Lightning session D: Rendering and JavaScript

    Blocking JavaScript or API resources in robots.txt is a classic rendering mistake: if Google's renderer cannot fetch a resource, it cannot run it.

47. Stage D2-C848 Day 2 · Welcome to indexing day!

    Google said it strongly believes site owners should be able to opt out of crawling and control how their site is crawled, which can matter for legal reasons or for crawl budget.

    repeats
    Stage D1-C329 Day 1 · How crawling works

    Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.

48. Stage D2-C850 Day 2 · Controlling indexing

    John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.

    repeats
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

49. Stage D2-C851 Day 2 · Controlling indexing

    When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.

    repeats
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

50. Stage D2-C851 Day 2 · Controlling indexing

    When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.

    repeats
    Slide D2-C021 Day 2 · Welcome to indexing day!

    Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.

Built on these claims 22

Developer requirements 21

• MustDocumentedDEV-SRV-05

  Rely only on the fields Google supports (user-agent, allow, disallow, sitemap) and keep robots.txt under 500 KiB

  Rests on 20 claims, 19 of them in this topic

• MustDocumentedDEV-IDX-01

  Use noindex to keep a page out of Search, and leave that URL crawlable

  Rests on 12 claims, 10 of them in this topic

• MayDocumentedDEV-AIF-05

  Set robots.txt policy per AI crawler by its own token, and know that user-triggered fetchers ignore it

  Rests on 13 claims, 6 of them in this topic

• AvoidDocumentedDEV-REN-04

  Do not block JavaScript, CSS or API endpoints that pages need for rendering in robots.txt

  Rests on 7 claims, 5 of them in this topic

• AvoidDocumentedDEV-CAN-11

  Never route a crawlable URL's redirects through a URL that robots.txt blocks

  Rests on 4 claims, 4 of them in this topic

• MustDocumentedDEV-SRV-04

  Keep robots.txt reachable on every host: 200 with rules, or 404 if there are none

  Rests on 4 claims, 4 of them in this topic

• MustDocumentedDEV-SRV-10

  Give a crawler its own robots.txt group only when needed, and make that group complete

  Rests on 5 claims, 4 of them in this topic

• ShouldDocumentedDEV-URL-05

  Publish XML sitemaps listing only canonical, indexable URLs with accurate lastmod

  Rests on 11 claims, 3 of them in this topic

• ShouldDocumentedDEV-IMG-06

  Keep images out of Search with robots.txt or X-Robots-Tag, not with CSS tricks

  Rests on 5 claims, 2 of them in this topic

• ShouldDocumentedDEV-SHP-02

  Let Storebot-Google crawl product, cart and checkout pages, through robots.txt and bot protection

  Rests on 6 claims, 2 of them in this topic

• AvoidDocumentedDEV-SRV-06

  Do not put Content-Signal or other non-standard lines between user-agent lines

  Rests on 6 claims, 2 of them in this topic

• MustDocumentedDEV-URL-08

  Keep faceted navigation out of the crawl with robots.txt rules or URL fragments

  Rests on 12 claims, 2 of them in this topic

• ShouldDocumentedDEV-VID-06

  Serve videos fast in a common container with standard codecs, and never block the files

  Rests on 8 claims, 2 of them in this topic

• MustDocumentedDEV-CAN-04

  Point every canonical directly at a final URL that returns 200, is indexable and is linked

  Rests on 8 claims, 1 of them in this topic

• AvoidDocumentedDEV-IDX-02

  Do not use noindex, nofollow or crawl-delay to save crawl budget

  Rests on 6 claims, 1 of them in this topic

• ShouldDocumentedDEV-IDX-03

  Send robots rules for PDFs, images and other non-HTML files in the X-Robots-Tag header

  Rests on 3 claims, 1 of them in this topic

• MayDocumentedDEV-IDX-11

  Use the Google-Extended robots.txt token to control Gemini training and grounding

  Rests on 8 claims, 1 of them in this topic

• MustDocumentedDEV-MON-02

  Render every template in a headless browser before release and check it in URL Inspection right after

  Rests on 7 claims, 1 of them in this topic

• ShouldDocumentedDEV-MON-10

  Plan releases and fixes around Google's processing times, and judge results only after them

  Rests on 11 claims, 1 of them in this topic

• ShouldDocumentedDEV-REN-09

  Monitor JavaScript console errors and Content Security Policy violations on key templates

  Rests on 4 claims, 1 of them in this topic

• AvoidDocumentedDEV-SRV-11

  Never use robots.txt to hide private paths; protect them with authentication

  Rests on 1 claim, 1 of them in this topic

Facts 1

• DocumentedF-025

  noindex costs crawl budget because Google must fetch a page to see the rule, while URLs disallowed in robots.txt are not fetched at all.

  Rests on 2 claims, 1 of them in this topic

Sources 27

• List of Google user-triggered fetchers Google · checked 3 October 2026
• In-depth guide to how Google Search works Google Search Central · checked 3 October 2026
• Overview of Google crawlers and fetchers (user agents) Google · checked 3 October 2026
• List of Google's common crawlers Google · checked 4 October 2026
• List of Google's special-case crawlers Google · checked 3 October 2026
• How Google interprets the robots.txt specification Google · checked 4 October 2026
• robots.txt report Google Search Console Help · checked 4 October 2026
• Formalizing the Robots Exclusion Protocol Specification Search Central blog (1 July 2019) · checked 4 October 2026
• Robots Refresher: Future-proof Robots Exclusion Protocol Search Central blog (28 March 2025) · checked 4 October 2026
• Introduction to robots.txt Google Search Central · checked 4 October 2026
• google/robotstxt Google (open-source robots.txt parser and matcher library, GitHub) · checked 4 October 2026
• URL Inspection tool Google Search Console Help · checked 4 October 2026
• Optimize your crawl budget Google · checked 4 October 2026
• No page information in search results Google Search Console Help · checked 2 October 2026
• Block Search indexing with noindex Google Search Central · checked 3 October 2026
• Build and submit a sitemap Google Search Central · checked 3 October 2026
• How to write and submit a robots.txt file Google Search Central · checked 3 October 2026
• Robots meta tag, data-nosnippet, and X-Robots-Tag specifications Google Search Central · checked 3 October 2026
• Crawling December: The how and why of Googlebot crawling Search Central blog (3 December 2024) · checked 2 October 2026
• Understand the JavaScript SEO basics Google Search Central · checked 3 October 2026
• Fix Search-related JavaScript problems Google Search Central · checked 2 October 2026
• Inside Googlebot: demystifying crawling, fetching, and the bytes we process Search Central blog (31 March 2026) · checked 3 October 2026
• 5 common mistakes with rel=canonical Search Central blog (8 April 2013) · checked 3 October 2026
• How to specify a canonical URL with rel="canonical" and other methods Google Search Central · checked 3 October 2026
• Remove images hosted on your site from search results Google Search Central · checked 3 October 2026
• Video SEO best practices Google Search Central · checked 3 October 2026
• Update your robots.txt file Google Search Central · checked 3 October 2026