Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Day 2 · Thursday 1 October 2026 · 10:30
Named on the title slide. The first recording stops mid-sentence at the closing slide; a second, independent recording covers the end of the talk (robots.txt and robots meta rules, robots meta tags and JavaScript).
John Mueller opened with a true-or-false quiz slide asking whether robots meta tags can make a page more visible in Search results than having none, and later answered that the statement is true.
“You can use robots meta tags to be more visible in Search results than without robots meta tags.”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-04
The robots meta tag is a piece of HTML code in the head section of a page that gives search engine crawlers page-specific instructions.
Speaker John MuellerEvidence slide photo, transcript
Used byglossary term Robots meta tag
The primary function of the robots meta tag is to control how a page is indexed and shown in search results.
Speaker John MuellerEvidence slide photo, transcript
Used byglossary term Robots meta tag
The robots meta tag is written as <meta name="robots" content="rule1,rule2">, or with the name googlebot instead of robots, with several rules separated by commas.
Speaker John MuellerEvidence slide photo
Used byglossary term Robots meta tag
The robots meta rule all is the default value and does nothing: it places no restrictions on indexing the page or following its links, the same as having no robots meta tag, and it is not an instruction that search engines must index the page.
“This is the default value - it does nothing”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-06
The noindex robots rule tells Google not to show the page in search results.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-01glossary term noindex
A slide named internal admin pages, temporary landing pages and thin content as typical use cases for the noindex rule.
Speaker John MuellerEvidence slide photo
The nosnippet rule stops Google from showing a text snippet or video preview for a page in search results, while the page's title is still shown.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-05glossary term nosnippet and data-nosnippet
The nosnippet rule also prevents a page's content from being used as a direct input for AI Overviews and AI Mode in Search.
“Also prevents the content from being used as a direct input for AI Overviews and AI Mode in Search results.”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-05glossary term nosnippet and data-nosnippet
The max-snippet:[number] rule limits a page's search result snippet to that number of characters (100 characters, for example), and max-snippet:0 shows no snippet, the same as nosnippet.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-04glossary term max-snippet
max-snippet:-1 removes the length limit and can produce a longer snippet than having no rule, because by default Google keeps snippets to a length it considers reasonable instead of quoting a page at length.
“max-snippet:-1 = No limit. (can be more than without)”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
The max-image-preview rule sets the maximum size of a page's image previews: none shows no preview, standard a default-sized one and large the largest possible, for example in Discover.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-04glossary term max-image-preview
The max-video-preview rule sets the maximum number of seconds of a video preview in Search; 0 allows only a static image and -1 means no time limit.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-VID-05
John Mueller's slide said Search Console has very few indexing-like settings and listed the Search generative AI control as the one control in his talk that is not a meta tag.
“Very few "indexing-like" settings are in Search Console”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
Setting the Search generative AI control in Search Console to exclude keeps a site's links and content out of Search generative AI features such as AI Overviews and AI Mode, so the site gets no traffic or impressions from them; include is the default.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-IDX-10
To be as visible as possible in Google, John Mueller's closing slide recommended the robots rules max-image-preview:large and max-snippet:-1.
“To be as visible as possible in Google, use: max-image-preview:large, max-snippet:-1”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, video, transcript
Used byrequirement DEV-IDX-04
A slide said robots meta rules can be added to a page's HTML with JavaScript, but doing so takes more time.
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-REN-05
Removing a robots restriction such as noindex with JavaScript does not work, a slide said.
“But... it takes more time, and removing restrictions (like "noindex") doesn't work.”
Wording checked against the slide or recording
Speaker John MuellerEvidence slide photo, transcript
Used byrequirement DEV-REN-05
John Mueller noted that robots.txt is defined by an internet standard, RFC 9309, which Google's Gary Illyes co-authored.
Speaker John MuellerEvidence transcript
John Mueller said there is currently no RFC that standardises robots meta tags, unlike robots.txt.
Speaker John MuellerEvidence transcript
John Mueller encouraged attendees to help write a specification for robots meta tags, saying internet standards groups are made up of ordinary people and need only passion and a willingness to join the discussions.
Speaker John MuellerEvidence transcript
John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.
Speaker John MuellerEvidence transcript
Used byglossary term robots.txt
The page-level nofollow robots rule tells search engines not to pass signals to any of the links on the page, which John Mueller called a weird and very broad rule that makes the page stand on its own.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-06glossary term nofollow
John Mueller recommends rel=nofollow on individual links instead of the page-level nofollow robots rule, so a site can choose which links are useful and which are not.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-06
John Mueller suspected that if robots meta tags were reinvented today, the page-level nofollow rule would probably not be part of them.
Speaker John MuellerEvidence transcript
John Mueller said AI crawlers do not really know what to do with nofollow links, because they look at the content rather than building a link graph; he did not say whether he meant Google's AI systems, other AI crawlers or both.
“AI crawlers don't really know what to do with a nofollow link, because they're looking at the content”
Speaker John MuellerEvidence transcript
The robots rule none is equivalent to noindex plus nofollow, so a page that carries it will not show up in Search, provided Google can see the tag.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-06
John Mueller said Google uses the snippet as a way of building AI Overviews and AI Mode answers, so if a page forbids a snippet, Google cannot use that snippet for them.
“we use the snippet as a way of building out the AI Overviews and the AI Mode answers”
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-05
John Mueller said data-nosnippet is rarely needed but lets a site keep a specific piece of text, such as a business phone number, out of the snippet, so the page can still be found for it while searchers have to visit the page to see it.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-07glossary term nosnippet and data-nosnippet
The data-nosnippet attribute can be used on only a handful of HTML elements.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-07
Broken HTML can widen data-nosnippet: if a div marked data-nosnippet is not closed properly, the rest of the page can be blocked from the snippet as well.
Speaker John MuellerEvidence transcript
Used byrequirements DEV-HTM-05, DEV-IDX-07
John Mueller said valid HTML is technically not a ranking factor but does matter for controls such as data-nosnippet.
“valid HTML is technically not an SEO ranking factor, but it does play a role”
Speaker John MuellerEvidence transcript
Used byrequirements DEV-HTM-05, DEV-IDX-07
John Mueller said, hedging with 'I think', that the max-snippet limit also applies to AI Overviews and AI Mode.
Speaker John MuellerEvidence transcript
A max-snippet value too short for a useful snippet may lead Google to show no snippet at all, John Mueller said.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-05
John Mueller said Google Search does not always show an image thumbnail for a result, and when it does, the thumbnail is usually limited to the size of the result entry.
Speaker John MuellerEvidence transcript
max-image-preview:large matters mainly in Discover, where it allows a large image that draws people's attention, so the rule can make a page more visible than leaving it out, John Mueller said.
Speaker John MuellerEvidence transcript
John Mueller said he thinks setting max-video-preview to -1 will not make Google show an hour-long video preview in search results.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-VID-05
John Mueller speculated that the use of max-video-preview may change in future, for example in Discover, where letting people view a full video could make sense.
Speaker John MuellerEvidence transcript
The notranslate robots rule tells Google not to offer a translation of the page in search results.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-09
John Mueller said notranslate can also be set in a meta tag named google, and that in that form it also turns off Chrome's automatic translation of the page.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-09
John Mueller said he dislikes notranslate because automatic translation is a form of accessibility for visitors who cannot read the page's language.
Speaker John MuellerEvidence transcript
The noimageindex rule tells Google not to index any of the images on the page, and John Mueller said he could not see why a site would want that.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-09
The unavailable_after rule lets a page drop out of search results after a set date and time, which suits time-bound pages, though John Mueller said most sites do not use it.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-08glossary term unavailable_after
John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-01
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-IDX-01
When Google finds a noindex rule in a page's HTML, it drops the page without even processing its JavaScript, so a script cannot switch the page back to indexable, John Mueller said.
“we will see the noindex and say, oh, we will get rid of this page; we won't even process the JavaScript”
Speaker John MuellerEvidence transcript
Used byrequirement DEV-REN-05
John Mueller recommended putting robots meta tags directly in a page's HTML exactly as intended, and adding them with JavaScript only where that is not possible, as in a JavaScript web app.
Speaker John MuellerEvidence transcript
Used byrequirement DEV-REN-05
Google's robots meta tag specification tells site owners to place the tag in the head section but notes that Google Search does not enforce that placement and also respects robots meta tags in the body of an HTML document.
Publisher Google Search Central
Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.
Publisher Google Search Central
Used byrequirements DEV-IDX-01, DEV-IDX-03glossary term X-Robots-Tag
Google's robots meta tag specification says that with nosnippet a static image thumbnail, if available, may still be shown when that results in a better user experience.
Publisher Google Search Central
Google's AI features guide says a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, with no additional technical requirements.
Publisher Google Search Central
Used byrequirements DEV-AIF-01, DEV-IDX-05glossary term AI Overviews
Google's robots meta tag specification says data-nosnippet may be extracted both before and after rendering, so the attribute should not be added to or removed from existing elements with JavaScript.
Publisher Google Search Central
Used byrequirements DEV-IDX-07, DEV-REN-05
Google's robots meta tag specification says Google chooses the snippet length when no max-snippet rule is set, and with max-snippet:-1 chooses the length it believes most effective; it does not say that -1 produces longer snippets.
Publisher Google Search Central
Used byrequirement DEV-IDX-04
Google's Search documentation gives notranslate as a rule for a robots or googlebot meta tag or an X-Robots-Tag header, and says the rule opts a page out of all translation features in Google Search.
Publisher Google Search Central
Used byrequirements DEV-IDX-03, DEV-IDX-09glossary term X-Robots-Tag
A 2008 Search Central blog post says a meta tag named google with the value notranslate stops Google Translate from translating any of a page's content.
Publisher Search Central blog (14 October 2008)
Used byrequirement DEV-IDX-09
Google's robots meta tag specification says Googlebot considerably decreases the crawl rate of a URL after the date and time set in its unavailable_after rule.
Publisher Google Search Central
Used byrequirement DEV-IDX-08
Google's meta tags documentation recommends avoiding JavaScript to inject or change meta tags whenever possible and testing the implementation thoroughly when it must be used.
Publisher Google Search Central
Used byrequirement DEV-REN-05
Google's JavaScript SEO basics guide says that when Google encounters a noindex rule it may skip rendering and JavaScript execution, so using JavaScript to change or remove a noindex robots meta tag may not work as expected.
“it may skip rendering and JavaScript execution”
Publisher Google Search Central
Used byrequirement DEV-REN-05
Audit robots meta tags set by templates and plug-ins: an explicit all rule does nothing and can go, while a page-level nofollow strips link signals from every link on the page and is better replaced by qualifying only the specific links that need it.
Author Ibrahim Anjro
Used byrequirement DEV-IDX-06
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Author Ibrahim Anjro
Before adding nosnippet or a low max-snippet value, weigh the cost in AI features: both limit how much of the page AI Overviews and AI Mode can use, and John Mueller described those answers as built from snippets.
Author Ibrahim Anjro
To hide one detail, such as a phone number, a price or a direct answer, instead of the whole snippet, wrap it in data-nosnippet on a span, div or section in the server HTML and validate the HTML so that an unclosed element cannot hide the rest of the page from snippets.
Author Ibrahim Anjro
Used byrequirement DEV-IDX-07
International sites should remove notranslate from their templates unless translation must be prevented: the robots or googlebot form switches off Google Search's translation features, and John Mueller said the form named google also blocks Chrome's translation for visitors who do not read the language.
Author Ibrahim Anjro
Used byrequirement DEV-IDX-09
Set max-snippet:-1 and max-image-preview:large on every indexable template unless licensing requires otherwise, and check that no CMS, plug-in or CDN setting adds lower snippet or image preview limits by default.
Author Ibrahim Anjro
Used byrequirement DEV-IDX-04
Ship robots rules in the HTML the server sends and never rely on JavaScript to lift a noindex: Google may skip rendering a page that arrives with noindex, so the page can stay out of the index even if a script removes the tag later.
Author Ibrahim Anjro
Used byrequirement DEV-REN-05
On stage the rule was absolute (Google won't even process the JavaScript of a page served with noindex), while Google's guide says only that it may skip rendering; either way, a noindex in the served HTML must never be one that JavaScript is expected to lift.
Author Ibrahim Anjro
John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Google treats nofollow as a hint for crawling, not a block: nofollow links will generally not be followed, but the linked pages may still be crawled if Google finds them through sitemaps or other links. To stop Google fetching URLs on your own site, use a robots.txt disallow rule.
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.
The noindex rule consumes crawl budget, because Google must fetch the page to see it.
The nosnippet rule also prevents a page's content from being used as a direct input for AI Overviews and AI Mode in Search.
Lead-generation, local-service and e-commerce sites should normally stay included, because AI answers cite and link sources. Sites selling paywalled or licensed content should test exclusion on a child property first. Use nosnippet only if you also want out of regular snippets.
John Mueller said Google uses the snippet as a way of building AI Overviews and AI Mode answers, so if a page forbids a snippet, Google cannot use that snippet for them.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Google's AI features guide says a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, with no additional technical requirements.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Setting the Search generative AI control in Search Console to exclude keeps a site's links and content out of Search generative AI features such as AI Overviews and AI Mode, so the site gets no traffic or impressions from them; include is the default.
Search Console has a property setting called Search generative AI that gives direct control over AI Overviews and AI Mode without affecting web rankings. Its states are inherit from parent, include and exclude.
Index selection checks for the robots unavailable_after rule and its date, and drops a page from the index once that date is reached.
The unavailable_after rule lets a page drop out of search results after a set date and time, which suits time-bound pages, though John Mueller said most sites do not use it.
Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).
John Mueller said Google uses the snippet as a way of building AI Overviews and AI Mode answers, so if a page forbids a snippet, Google cannot use that snippet for them.
Google's Discover slide recommended large images at least 1,200 px wide, with more than 300,000 total pixels and a 16x9 aspect ratio, enabled by the max-image-preview:large setting.
max-image-preview:large matters mainly in Discover, where it allows a large image that draws people's attention, so the rule can make a page more visible than leaving it out, John Mueller said.
John Mueller noted that robots.txt is defined by an internet standard, RFC 9309, which Google's Gary Illyes co-authored.
Robots.txt is now an IETF standard, RFC 9309, supported by basically all major crawlers; Google said it follows it because it is the right thing to do: anyone who wants to opt out of crawling should be able to.
Setting the Search generative AI control in Search Console to exclude keeps a site's links and content out of Search generative AI features such as AI Overviews and AI Mode, so the site gets no traffic or impressions from them; include is the default.
The Search generative AI control covers AI Overviews, AI Mode and generative AI features in Discover. Excluding removes both links to the site and use of its content to ground answers.
Natalia Venditto opened by saying that Gary had said earlier at the event that everyone can write a standard, and that this is what the Web Fragments work is trying to do.
John Mueller encouraged attendees to help write a specification for robots meta tags, saying internet standards groups are made up of ordinary people and need only passion and a willingness to join the discussions.
John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.
When robots.txt blocks a page, the page's content is not indexable but Google can still index its URL, John Mueller said.
Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will still find the URLs of those pages in Search.
The noimageindex robots meta tag tells Google not to index the images on the page.
The noimageindex rule tells Google not to index any of the images on the page, and John Mueller said he could not see why a site would want that.
Setting the max-image-preview robots meta tag to large can make content perform surprisingly well in Discover, Gary Illyes said.
max-image-preview:large matters mainly in Discover, where it allows a large image that draws people's attention, so the rule can make a page more visible than leaving it out, John Mueller said.