Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Thing · Standard
The text file at the root of a host that tells crawlers which paths they may fetch, standardised by the IETF in 2022 as the Robots Exclusion Protocol (RFC 9309).
Glossary · robots.txt
A file in the root of each host (example.com/robots.txt) that tells crawlers which URLs they may fetch. It controls crawling, not indexing, applies only to its own host and is not a security measure: anyone can read it.
Documented in
Google's crawler documentation says special-case crawlers serve specific Google products where the crawled site and the product have an agreement about the crawl process, so they may ignore robots.txt rules; AdsBot, for example, ignores the global (*) user agent with the ad publisher's permission.
Google · Day 1 · How crawling works
If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the cached copy for up to 30 days.
In robots.txt, * matches zero or more of any character and $ marks the end of the URL.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google generally caches robots.txt for up to 24 hours and reads only the first 500 KiB of the file.
Search Console's robots.txt report shows the robots.txt files Google found for the top 20 hosts of a Domain property, with their fetch status, warnings and errors; to test whether a specific URL is blocked, Google's help page points to the URL Inspection tool and to Google's open-source robots.txt library.
Google Search Console Help · Day 1 · How Google interprets robots.txt
Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt and keep crawlable only item pages plus one unfiltered listing page, or use URL fragments, which Google generally does not crawl. rel=canonical and nofollow are weaker, slower options.
Google treats nofollow as a hint for crawling, not a block: nofollow links will generally not be followed, but the linked pages may still be crawled if Google finds them through sitemaps or other links. To stop Google fetching URLs on your own site, use a robots.txt disallow rule.
Google Search Central, Search Central blog (10 September 2019) · Day 1 · How Google thinks about crawl budget
Google advises against using noindex to save crawl budget and against using robots.txt to temporarily reallocate budget. Use robots.txt only for pages you never want crawled, and 404 or 410 for removed pages.
Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.
Google Search Central · Day 2 · Welcome to indexing day!
Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.
Google Search Central · Day 2 · Controlling indexing
Google's crawler documentation defines grounding in Gemini Apps and in Grounding with Google Search on Vertex AI as providing content from the Google Search index to the model at prompt time, and sites manage whether their content is used for it with the Google-Extended robots.txt token.
Google · Day 2 · Lightning session D: Rendering and JavaScript
Google says user-triggered fetchers, which fetch a URL because a user asked for it in a Google product (for example Gemini Notebook fetching URLs users add as sources, or Google-Agent acting on a user's request), generally ignore robots.txt rules.
Google · Day 2 · Lightning session D: Rendering and JavaScript
Google's introduction to robots.txt says unimportant image, script or style files may be blocked, but resources whose absence makes a page harder for Google to understand should not be blocked.
Google Search Central · Day 2 · What is Google friendly JavaScript
Google's documented ways to keep a site's images out of search results are a robots.txt disallow rule (for example for Googlebot-Image) or a noindex X-Robots-Tag HTTP header, with the Removals tool for emergencies.
Google Search Central · Day 2 · Using images to your advantage and Engaging Search users with videos
Google's crawler list says robots.txt rules addressed to the Storebot-Google user agent affect all surfaces of Google Shopping, such as the Shopping tab in Google Search.
Google's robots.txt guide says its crawlers update their cached copy of a site's robots.txt every 24 hours, and that the Request a recrawl option in Search Console's robots.txt report refreshes it faster.
Google Search Central, Google Search Console Help · Day 3 · How long does it take to..?
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Bing closes the robots.txt group at an unknown line placed between two user-agent lines, so in the slide's example bingbot ends up not blocked while Googlebot is; the slide warned that different crawlers behave differently.
The robots.txt talk pointed to the robots.txt file of thebestfriedchickenever.com to show why comments in robots.txt are useful; the live file is mostly a large ASCII-art drawing written as # comment lines, followed by a single user-agent group (checked 2026-10-04).
Other companies have similar 'extended' robots.txt tokens; Google named Apple's Applebot-Extended.
Google called robots.txt extremely forgiving: a typo in a path only blocks the wrong path, a typo in a rule name such as disallow makes Google ignore that line, and the rest of the file is still used.
The real trouble is an unknown line placed between two user-agent lines: while writing RFC 9309, Google asked people whether the first crawler should then inherit the rules that follow, and opinions split roughly 50-50 (how the talk said the question was settled is unclear in the recording).
Google said the robots.txt report helps catch CDNs that update a site's robots.txt without the owner's knowledge, and hosts that cloak the robots.txt file, which happens more often than people think.
Dave Smart said robots.txt is checked for every URL in a redirect chain and crawling stops at the first blocked one; in his example a site redirected through /cart/ with JavaScript to set the local currency and back, and because /cart/ was disallowed the page was reported as blocked.
Dave Smart · Day 1 · Lightning session B: Robots.txt
Dave Smart said this applies to all redirects, not only JavaScript ones; his examples: a redirect through an external authorisation service that is blocked by its own robots.txt, content that moved through several URLs over the years with one of them later blocked, and unexpected redirects, such as one served only to Googlebot's user agent.
Dave Smart · Day 1 · Lightning session B: Robots.txt
A Google panelist said they were working on a set of crawler best practices and offering research, malware-scanning and privacy crawlers an exemption from following them, apparently because such crawlers sometimes need to ignore robots.txt or probe URLs that other crawlers would not touch (the reason is a best reading: the audio has 'don't need', which would not explain an exemption).
A Google panelist said Apple offers a robots.txt control for Applebot similar to Google-Extended, so a site can allow Applebot for search while opting out of Apple's AI use.
When logs show an unknown crawler fetching a lot, search for its user agent online to find the robots.txt token that controls it; the mainstream crawlers that send the most traffic can all be controlled this way.
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site, such as a national tax authority, that blocks very important PDF files with robots.txt: Google cannot index the PDFs' content but can at least show their URLs in search results.
Google said very few URLs disallowed by robots.txt are in its index, compared with the index as a whole (no figure given).
Google's two tokenization slides showed the difference on the same sentence: the Search tokenizer kept 'robots.txt' and 'tl;dr' as single tokens, while the AI-model tokenizer split them into pieces such as 'tl' and 'dr' or 'robots' and 'txt', with the punctuation as separate tokens.
Gary Illyes · Day 2 · Understanding what's on a page
Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt is not shown by its URL, because Google sees no good reason to show a video URL alone.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.
Gary Illyes · Day 1 · How crawling works
Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.
Gary Illyes · Day 1 · How crawling works
The only Google-owned crawlers that do not obey robots.txt are contractual crawlers, which crawl a site whose owner has agreed that Google may crawl it however it likes.
Gary Illyes · Day 1 · How crawling works
Google said it already supported robots.txt when it started crawling in 1996 as a Stanford PhD project, and has supported it ever since.
The standard's official name is the Robots Exclusion Protocol (REP). Google stressed that it was designed in 1994 only to control which automated clients may access what on a site, and has nothing to do with how the content is used.
Robots.txt is not a security measure: the file sits at a predictable public address, so anyone can read the paths it disallows. Protect a secret folder with authentication, or do not put it on the internet.
Robots.txt was made extremely simple by design so that anyone could implement and understand it; it was extended as websites grew more complex, but it still has only three rules: user-agent (a named crawler, or * for every crawler), disallow and allow.
Google said a site can keep a directory or the whole site out of Gemini model training by disallowing it for the Google-Extended token in robots.txt, and that Google respects that policy; the speaker believed Google was the first to offer such an opt-out from model training.
Dave Smart said Search Console reports such a block only on the first URL of the chain, which is confusing: the URL shown as blocked by robots.txt is not itself disallowed.
Dave Smart · Day 1 · Lightning session B: Robots.txt
When Search Console says a URL is blocked by robots.txt but the file does not block it, check whether the URL redirects and test every URL in the chain.
Dave Smart · Day 1 · Lightning session B: Robots.txt
URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
Google follows robots.txt partly in its own interest: crawling an infinite URL space such as a calendar that robots.txt blocks would waste Google's crawling time as well as the site's resources.
AI agents acting on a user's request fall into the same category as user-initiated fetchers: an agent told to look at a site reads the page without checking robots.txt.
Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.
Gary Illyes · Day 1 · Q&A
Nearly all mainstream AI crawlers and AI systems use their own user agents, so robots.txt can set a separate policy for each; at the event OpenAI was said to have three or four, Anthropic a few and Microsoft some.
Google's Q&A slide said that a sitemap not listed in robots.txt has to be submitted instead, for example in Search Console.
Google pointed out the trade-off of listing a sitemap in robots.txt: anyone can then see the sitemap, which a site may not want.
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
John Mueller said there is currently no RFC that standardises robots meta tags, unlike robots.txt.
John Mueller · Day 2 · Controlling indexing
For each suspect script or API request in Chrome DevTools, check whether it runs, whether something blocks it and whether robots.txt disallows it.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
Disallowing a script folder such as /static/js/ or a generic /api/ folder in robots.txt can block the endpoints that supply a page's content, leaving blank modules and missing content after rendering.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
If an API folder must stay blocked in robots.txt, allow the endpoints that rendering needs, for example Disallow: /api/ together with Allow: /api/products/.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
Blocked resources, one of Google's four common JavaScript indexing problems, means robots.txt disallowing the crawling of critical JavaScript files or API endpoints.
Erin Sparling · Day 2 · What is Google friendly JavaScript
The canonical leader of a group should be indexable: it should carry no noindex robots directive, return no error status code and not be blocked in robots.txt.
Tobias Schwarz · Day 2 · Lightning session E: Managing Duplicates and Site Moves
A community speaker pointed out that, according to Google's documentation, user-triggered fetchers, which fetch a page because a user asked for it, generally ignore robots.txt rules.
Carlos Ortega · Day 1 · Lightning session A: Automation and AI
Google's robots.txt talk recalled that robots.txt began in 1994, when bots were crashing servers: Martijn Koster proposed a text file in the root of a site with rules for how automated clients may access it.
Robots.txt is now an IETF standard, RFC 9309, supported by basically all major crawlers; Google said it follows it because it is the right thing to do: anyone who wants to opt out of crawling should be able to.
The robots.txt file always sits in the root of the host; it cannot be placed anywhere else.
Comments in robots.txt start with #, and a line without the # that a parser cannot read is ignored anyway, because the standard requires parsers to skip lines they cannot parse, so it acts like a comment.
Search Console's robots.txt report shows the robots.txt file as Google last fetched it, with a version history and the errors and successes of each fetch.
Dave Smart said robots.txt groups are not additive: if a group matches a crawler's user agent, only that group's rules apply, so a googlebot group that blocks /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks.
Dave Smart · Day 1 · Lightning session B: Robots.txt
Mainstream crawlers from Google, other large search engines and AI companies try to follow robots.txt, so implementing robots.txt correctly is the way to stop them doing something specific on a site.
User-initiated fetchers, such as a translation service fetching a page a user asked to translate, are a different category from crawlers and generally do not check robots.txt, because a user asked for the fetch.
To opt out of Google's AI training, a site can use the Google-Extended token in robots.txt.
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
Google said listing the sitemap in robots.txt is fine, as many websites do.
John Mueller noted that robots.txt is defined by an internet standard, RFC 9309, which Google's Gary Illyes co-authored.
John Mueller · Day 2 · Controlling indexing
John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.
John Mueller · Day 2 · Controlling indexing
John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.
John Mueller · Day 2 · Controlling indexing
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
John Mueller · Day 2 · Controlling indexing
Never block a resource that a page needs for rendering with robots.txt, a community speaker said, calling this common sense.
Sören Bendig · Day 2 · Lightning session D: Rendering and JavaScript
Blocking JavaScript or API resources in robots.txt is a classic rendering mistake: if Google's renderer cannot fetch a resource, it cannot run it.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
robots.txt rules apply per host, so the rules in example.com/robots.txt do not apply at all to an API served from api.example.com.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
If robots.txt blocks JavaScript that client-side code needs to render, Google cannot render the content that the JavaScript would produce.
Erin Sparling · Day 2 · What is Google friendly JavaScript
Googlebot-Image and Googlebot-Video are the Google crawlers that fetch images and videos, and robots.txt rules addressed to their user agent tokens control how Google indexes a site's images and videos.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
If robots.txt disallows the location of an image or video file, Google does not index that image or video.
Gary Illyes · Day 2 · Using images to your advantage and Engaging Search users with videos
Google estimated that a robots.txt update is picked up in about 24 hours, with a minimum of seconds and an end point of 25 hours on the slide.
Google said its service level objective is to refresh robots.txt every 24 hours, as RFC 9309 calls for, though delays happen now and then.
Gary Illyes · Day 3 · How long does it take to..?
A site owner can submit robots.txt in Search Console to force Google to refresh it sooner.
Gary Illyes · Day 3 · How long does it take to..?
The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.
An audience question asked how to make sure bots behave well beyond robots.txt.
From the audience · Day 1 · Q&A
A Google panelist said crawlers that ignore robots.txt and cause a nuisance are better treated as a scraping problem than as a crawling problem.
A Google panelist argued that it makes business sense for agents not to follow robots.txt: an agent sent to buy something that obeyed a robots.txt block could not complete the purchase, a bad experience for the user and lost revenue for the shop.
A Google panelist said much of robots.txt is written with search engines in mind: a search engine should never add items to a cart or check out, but an agent acting for a user probably should be able to.
A Google panelist doubted that setting different robots.txt policies per AI crawler makes practical sense yet, because nobody knows how these systems will develop.
A Google panelist called a default-deny robots.txt, which blocks every crawler and allows only chosen ones, a bad pattern because the site owner does not know what is being blocked.
An audience member asked whether the sitemap link should be included in robots.txt, noting that many websites include it but that it was missing from an earlier robots.txt slide by a Google speaker, which the question called 'Gary's slide'.
From the audience · Day 2 · Welcome to indexing day!
An audience member noted that a robots.txt block does not mean noindex and that explaining this to developers is a constant struggle, and asked why Google does not treat a robots.txt block as a noindex.
From the audience · Day 2 · Welcome to indexing day!
A community speaker listed four common but often overlooked JavaScript rendering mistakes: non-crawlable link markup, error pages that return HTTP 200 in single-page apps, content behind user interaction, and JavaScript or API resources blocked in robots.txt.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
In the speaker's product page example, three of the mistakes combine: product links behind onclick handlers keep the product detail pages hidden, the product API is blocked in robots.txt, and some content waits for a user interaction.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
Content-Signal lines are now common because a large CDN provider adds them to its managed robots.txt files. Never place any non-standard directive between user-agent lines; put it after a group's rules and test the file with each search engine's tools, such as Bing's robots.txt tester and Google's open-source robots.txt parser.
Ibrahim Anjro · Day 1 · How Google interprets robots.txt
The 'first' needs care: OpenAI documented how to block its GPTBot crawler in robots.txt in August 2023, before Google announced Google-Extended on 28 September 2023. What Google-Extended added was a token that controls AI-training use of pages fetched by Google's existing crawlers, separately from Search; Apple later introduced a similar token, Applebot-Extended.
Ibrahim Anjro · Day 1 · How Google interprets robots.txt
A typo in a rule name is not always ignored by Google: its open-source robots.txt parser deliberately accepts common misspellings of disallow (such as dissallow, dissalow and disalow) and of user-agent (useragent, user agent), but not of allow. Google's spec page does not mention typos, and other crawlers may be stricter, so spell rule names correctly.
Ibrahim Anjro · Day 1 · How Google interprets robots.txt
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
Ibrahim Anjro · Day 1 · Q&A
Explain robots.txt and noindex to developers as two separate controls: robots.txt controls crawling, noindex controls indexing. To keep a page out of Search, let Google crawl it and serve noindex; a robots.txt disallow alone can leave the bare URL in results.
Ibrahim Anjro · Day 2 · Welcome to indexing day!
Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.
Ibrahim Anjro · Day 2 · Welcome to indexing day!
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Ibrahim Anjro · Day 2 · Controlling indexing
If Gemini training reuses the rendering done for Search, a site needs no separate rendering work for Gemini; whether its rendered content is used for training is decided with the Google-Extended token in robots.txt, not by rendering choices.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
Google documents Gemini grounding only as content from the Search index at prompt time; the live read of a specific page at a user's request, described on stage, is not documented, so it is unclear whether it works like a user-triggered fetcher, which generally ignores robots.txt, or follows Google-Extended.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
When something is missing from a rendered page, check both robots.txt and the Content Security Policy: the URL Inspection live test shows the page resources, the JavaScript console output and a screenshot of the rendered page.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
A robots.txt carve-out works because Google applies the most specific matching rule, so Allow: /api/products/ beats Disallow: /api/ for product endpoints only; re-test a rendered page after every robots.txt change to script or API paths.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
An API or CDN on its own host needs its own robots.txt check: a blanket Disallow there, or a robots.txt that returns 5xx errors, can stop Google fetching the data a page renders from.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
Make sure robots.txt does not block the URLs from which the browser loads a fragment's assets (the gateway paths on the host page's origin, or the endpoint's own host if assets load from there), because Google does not render JavaScript from blocked files and robots.txt rules apply per host.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
Check robots.txt for Disallow rules covering JavaScript bundles, build folders or API endpoints the page calls while rendering, including on separate API or CDN hostnames with their own robots.txt, then confirm in URL Inspection's live test that the rendered HTML contains the main content.
Ibrahim Anjro · Day 2 · What is Google friendly JavaScript
Hiding an image as a CSS background is a fragile way to keep it out of Google: it only stops extraction from that page, so the same image URL used in an img element elsewhere or listed in a sitemap can still be indexed; the documented robots.txt or noindex X-Robots-Tag methods are the reliable route.
Ibrahim Anjro · Day 2 · Using images to your advantage and Engaging Search users with videos
Treat Storebot-Google separately from Googlebot in robots.txt and bot protection: a shop that blocks it on product, cart or checkout paths can undermine Shopping price and availability checks, even when Googlebot is allowed.
Ibrahim Anjro · Day 3 · Shopping on Search: Beyond the blue links
Plan robots.txt changes a day ahead, because Google may use the old file for about 24 hours; for an urgent change, request a refresh in Search Console instead of waiting.
Ibrahim Anjro · Day 3 · How long does it take to..?
Kit items about robots.txt: their own words name it, or several of the claims they rest on do.
Rests on 20 claims, 14 of them naming robots.txt; its own words name robots.txt
Rests on 13 claims, 11 of them naming robots.txt; its own words name robots.txt
Use noindex to keep a page out of Search, and leave that URL crawlable
Rests on 12 claims, 11 of them naming robots.txt; its own words name robots.txt
Use the Google-Extended robots.txt token to control Gemini training and grounding
Rests on 8 claims, 5 of them naming robots.txt; its own words name robots.txt
Do not block JavaScript, CSS or API endpoints that pages need for rendering in robots.txt
Rests on 7 claims, 5 of them naming robots.txt; its own words name robots.txt
Do not put Content-Signal or other non-standard lines between user-agent lines
Rests on 6 claims, 5 of them naming robots.txt; its own words name robots.txt
Never route a crawlable URL's redirects through a URL that robots.txt blocks
Rests on 4 claims, 4 of them naming robots.txt; its own words name robots.txt
Keep images out of Search with robots.txt or X-Robots-Tag, not with CSS tricks
Rests on 5 claims, 4 of them naming robots.txt; its own words name robots.txt
Keep robots.txt reachable on every host: 200 with rules, or 404 if there are none
Rests on 4 claims, 4 of them naming robots.txt; its own words name robots.txt
Publish XML sitemaps listing only canonical, indexable URLs with accurate lastmod
Rests on 11 claims, 4 of them naming robots.txt; its own words name robots.txt
Do not use noindex, nofollow or crawl-delay to save crawl budget
Rests on 6 claims, 3 of them naming robots.txt; its own words name robots.txt
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 3 of them naming robots.txt; its own words name robots.txt
Send robots rules for PDFs, images and other non-HTML files in the X-Robots-Tag header
Rests on 3 claims, 2 of them naming robots.txt
Let Storebot-Google crawl product, cart and checkout pages, through robots.txt and bot protection
Rests on 6 claims, 2 of them naming robots.txt; its own words name robots.txt
Serve videos fast in a common container with standard codecs, and never block the files
Rests on 8 claims, 2 of them naming robots.txt; its own words name robots.txt
Point every canonical directly at a final URL that returns 200, is indexable and is linked
Rests on 8 claims, 1 of them naming robots.txt; its own words name robots.txt
Plan releases and fixes around Google's processing times, and judge results only after them
Rests on 11 claims, 1 of them naming robots.txt; its own words name robots.txt
Give a crawler its own robots.txt group only when needed, and make that group complete
Rests on 5 claims, 1 of them naming robots.txt; its own words name robots.txt
Never use robots.txt to hide private paths; protect them with authentication
Rests on 1 claim, 1 of them naming robots.txt; its own words name robots.txt
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 0 of them naming robots.txt; its own words name robots.txt
Rests on 2 claims, 1 of them naming robots.txt; its own words name robots.txt
Also inglossary terms robots.txt, RFC 9309 (Robots Exclusion Protocol), User-triggered fetchers, Contractual crawlers (special-case crawlers), Googlebot-Image and Googlebot-Video, Redirect chain, X-Robots-Tag, Faceted navigation, Google-Extended, Grounding, nofollow, noindex, robots.txt report, Storebot-Google, User-agent group
Things named in the same claim, with the number of claims they share.