Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
Disallowed URLs are not fetched, so they cost no crawl budget. Pages with noindex must be fetched, so they do; nofollow is a hint, not a block, so the linked URLs can still be crawled, and crawl-delay is ignored by Googlebot. Day 2 stressed that robots.txt does not control indexing: a disallowed URL can still appear in results without its content, and a noindex on a blocked URL is never seen. John Mueller advised against the page-level nofollow robots rule, which stops signals passing through every link on the page, in favour of rel=nofollow on individual links. He also said AI crawlers do not really know what to do with nofollow, without saying whose crawlers he meant (not in Google's docs). A second recording of Day 1 explained the slide on nofollow: Google does not crawl through the nofollow link itself but still crawls the linked page when it finds it through other links, as Google's 2017 crawl budget post also says. A second recording of Day 2 confirmed John Mueller's remark that if robots meta tags were reinvented today the page-level nofollow rule would probably not be part of them. Day 2's opening Q&A illustrated why a disallow is not a noindex with a national tax authority that blocks very important PDFs, whose URLs Google can show without their content, and John Mueller said robots meta rules only work if robots.txt allows the fetch. Disallowing an image or video file keeps it out of Google, and a disallowed video is not shown by its bare URL. In the Day 1 Q&A Gary Illyes said disallowing a section such as /ads shifts crawl budget to the rest of the site. Author’s view: Google's guide says freed crawl budget shifts only on sites already at their capacity limit, so block only sections you never want crawled.
Things in this topic 19
Counts are claims that name the thing. All things
What to do
URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-IDX-02fact F-025
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-IDX-02fact F-025
The nofollow rule can still consume crawl budget: Google does not crawl through the nofollow link itself, but it still crawls the linked page when it finds that page through other links.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-IDX-02
The non-standard crawl-delay rule is not processed by Googlebot, so it does not save any crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirements DEV-IDX-02, DEV-SRV-05
Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.
Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Publisher GoogleAnnotates Day 1, 15:10 · How Google interprets robots.txt
Used byrequirement DEV-SRV-05glossary term robots.txt
Google treats nofollow as a hint for crawling, not a block: nofollow links will generally not be followed, but the linked pages may still be crawled if Google finds them through sitemaps or other links. To stop Google fetching URLs on your own site, use a robots.txt disallow rule.
Publisher Google Search Central, Search Central blog (10 September 2019)Annotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirements DEV-IDX-02, DEV-IDX-06glossary term nofollow
Google advises against using noindex to save crawl budget and against using robots.txt to temporarily reallocate budget. Use robots.txt only for pages you never want crawled, and 404 or 410 for removed pages.
Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirements DEV-ERR-01, DEV-IDX-02
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirement DEV-IDX-01
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
Author Ibrahim AnjroAnnotates Day 1, 16:35 · Q&A
An audience member noted that a robots.txt block does not mean noindex and that explaining this to developers is a constant struggle, and asked why Google does not treat a robots.txt block as a noindex.
From the audienceIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
“Simply put, it's because of sites that are extremely important and like to disallow their most important pages, either accidentally or out of ignorance.”
Wording checked against the slide or recording
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Used byrequirement DEV-IDX-01
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
Speaker Gary Illyes, Cherry PrommawinIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript
Used byrequirement DEV-IDX-01
Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site, such as a national tax authority, that blocks very important PDF files with robots.txt: Google cannot index the PDFs' content but can at least show their URLs in search results.
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
Used byrequirement DEV-IDX-01
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
“if a URL is important, then it might get indexed even if it's disallowed by robots.txt. So the URL gets indexed, not the content.”
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
Used byrequirement DEV-IDX-01
John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
Used byglossary term robots.txt
The page-level nofollow robots rule tells search engines not to pass signals to any of the links on the page, which John Mueller called a weird and very broad rule that makes the page stand on its own.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
Used byrequirement DEV-IDX-06glossary term nofollow
John Mueller recommends rel=nofollow on individual links instead of the page-level nofollow robots rule, so a site can choose which links are useful and which are not.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
Used byrequirement DEV-IDX-06
John Mueller suspected that if robots meta tags were reinvented today, the page-level nofollow rule would probably not be part of them.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
John Mueller said AI crawlers do not really know what to do with nofollow links, because they look at the content rather than building a link graph; he did not say whether he meant Google's AI systems, other AI crawlers or both.
“AI crawlers don't really know what to do with a nofollow link, because they're looking at the content”
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
Used byrequirement DEV-IDX-01
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript
Used byrequirement DEV-IDX-01
Gary Illyes suggested a div with a CSS background image as a way to keep an image from being picked up by Google.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirement DEV-IMG-06
If robots.txt disallows the location of an image or video file, Google does not index that image or video.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirements DEV-IMG-06, DEV-VID-06glossary term Googlebot-Image and Googlebot-Video
Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt is not shown by its URL, because Google sees no good reason to show a video URL alone.
Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript
Used byrequirement DEV-VID-06
Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.
Publisher Google Search CentralAnnotates Day 2, 10:15 · Welcome to indexing day!
Used byrequirement DEV-IDX-01glossary term noindex
Google's documented ways to keep a site's images out of search results are a robots.txt disallow rule (for example for Googlebot-Image) or a noindex X-Robots-Tag HTTP header, with the Removals tool for emergencies.
Publisher Google Search CentralAnnotates Day 2, 13:50 · Using images to your advantage and Engaging Search users with videos
Used byrequirements DEV-IDX-03, DEV-IMG-06glossary term X-Robots-Tag
Explain robots.txt and noindex to developers as two separate controls: robots.txt controls crawling, noindex controls indexing. To keep a page out of Search, let Google crawl it and serve noindex; a robots.txt disallow alone can leave the bare URL in results.
Author Ibrahim AnjroAnnotates Day 2, 10:15 · Welcome to indexing day!
Used byrequirement DEV-IDX-01
Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.
Author Ibrahim AnjroAnnotates Day 2, 10:15 · Welcome to indexing day!
Audit robots meta tags set by templates and plug-ins: an explicit all rule does nothing and can go, while a page-level nofollow strips link signals from every link on the page and is better replaced by qualifying only the specific links that need it.
Author Ibrahim AnjroAnnotates Day 2, 10:30 · Controlling indexing
Used byrequirement DEV-IDX-06
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Author Ibrahim AnjroAnnotates Day 2, 10:30 · Controlling indexing
Hiding an image as a CSS background is a fragile way to keep it out of Google: it only stops extraction from that page, so the same image URL used in an img element elsewhere or listed in a sitemap can still be indexed; the documented robots.txt or noindex X-Robots-Tag methods are the reliable route.
Author Ibrahim AnjroAnnotates Day 2, 13:50 · Using images to your advantage and Engaging Search users with videos
Used byrequirement DEV-IMG-06
Google's crawl budget guide says crawl budget freed by robots.txt blocks is not shifted to other pages unless the site already hits its crawl capacity limit, and advises against robots.txt for temporary reallocation; so block only sections you never want crawled, and expect a shift only on capacity-limited sites.
Robots.txt was made extremely simple by design so that anyone could implement and understand it; it was extended as websites grew more complex, but it still has only three rules: user-agent (a named crawler, or * for every crawler), disallow and allow.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google answered on a Q&A slide that a sitemap listed in the robots.txt file can be picked up by any crawler, not only by Google's.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
URLs disallowed through robots.txt do not affect crawl budget, because they are not fetched.
Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Google treats nofollow as a hint for crawling, not a block: nofollow links will generally not be followed, but the linked pages may still be crawled if Google finds them through sitemaps or other links. To stop Google fetching URLs on your own site, use a robots.txt disallow rule.
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
Index selection drops a document carrying a noindex rule if it was not already dropped earlier in processing (noindex is a likely but not certain reading of the transcript, supported by the later mention of noindex among the Page indexing report reasons).
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
John Mueller suspected that if robots meta tags were reinvented today, the page-level nofollow rule would probably not be part of them.
The page-level nofollow robots rule tells search engines not to pass signals to any of the links on the page, which John Mueller called a weird and very broad rule that makes the page stand on its own.
Google said listing the sitemap in robots.txt is fine, as many websites do.
Google supports only the user-agent, allow, disallow and sitemap fields in robots.txt. Other fields such as crawl-delay are not supported.
Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site, such as a national tax authority, that blocks very important PDF files with robots.txt: Google cannot index the PDFs' content but can at least show their URLs in search results.
Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some extremely important sites disallow their most important pages, by accident or out of ignorance.
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the URL is indexed but not its content.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
Google's robots.txt introduction names links from elsewhere on the web as the reason a disallowed URL can still be indexed, while on stage Google spoke of the URL's importance; either way, well-linked important URLs are the disallowed ones most likely to appear in results, so keep such pages crawlable with noindex if they must stay out of Search.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
Unlike a disallowed web page, whose bare URL can still appear in web results, a video blocked by robots.txt is not shown by its URL, because Google sees no good reason to show a video URL alone.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
Use noindex to keep a page out of Search, and leave that URL crawlable
Rests on 12 claims, 9 of them in this topic
Do not use noindex, nofollow or crawl-delay to save crawl budget
Rests on 6 claims, 6 of them in this topic
Prefer rel attributes on individual links to a page-level nofollow
Rests on 6 claims, 4 of them in this topic
Keep images out of Search with robots.txt or X-Robots-Tag, not with CSS tricks
Rests on 5 claims, 4 of them in this topic
Rests on 20 claims, 2 of them in this topic
Serve videos fast in a common container with standard codecs, and never block the files
Rests on 8 claims, 2 of them in this topic
Return 404 or 410 for removed and non-existent URLs
Rests on 7 claims, 1 of them in this topic
Send robots rules for PDFs, images and other non-HTML files in the X-Robots-Tag header
Rests on 3 claims, 1 of them in this topic
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 1 of them in this topic
Rests on 2 claims, 2 of them in this topic