Search Central LiveDeep Dive Europe 2026

Knowledge base v2.13.0 · Community edition · data through 2 October 2026

Topic · Publisher controls

Robots meta tags and noindex

Robots meta tags (name robots or googlebot, comma-separated rules, placed in the head, though Google also respects them in the body) are how site owners control indexing, because robots.txt controls only crawling; Google said it does not treat a robots.txt disallow as noindex because some very important sites disallow their key pages by accident or out of ignorance, and a noindex on a disallowed URL is never seen. John Mueller went through the rules: all is the default and does nothing (it is not an instruction to index), noindex keeps a page out of results, none equals noindex plus nofollow, page-level nofollow is a very broad rule he would replace with rel=nofollow on individual links, noimageindex blocks every image on the page, and unavailable_after drops a page after a set date, though most sites do not use it. Google said index selection drops a document carrying noindex (a likely but not certain reading of the recording) if it was not dropped earlier, and removes a page once its unavailable_after date passes, extending Day 1's point that Google must fetch a page to see its noindex. Robots rules added with JavaScript take longer to take effect and removing a noindex with JavaScript does not work, and Google's documentation advises avoiding JavaScript for meta tags. Unlike robots.txt, robots meta tags have no RFC, and John Mueller invited SEOs to help write a specification. Day 3 timed the rules: Google estimated that meta annotations such as robots meta tags are typically processed in 45 to 90 minutes (not in Google's docs), and that once Google has processed a page that now returns a 404 or noindex, the page usually leaves the serving index within one to three weeks, sometimes much sooner. A second recording of Day 2 covered the end of John Mueller's talk: robots meta rules only work if robots.txt lets Google fetch the page; when Google finds a noindex in the HTML it drops the page without even processing its JavaScript, so a script cannot make the page indexable again (Google's JavaScript guide says it may skip rendering), and he recommended putting robots meta tags in the HTML exactly as intended, using JavaScript only where that is impossible. He also suspected that the page-level nofollow rule would probably not be part of robots meta tags if they were reinvented today. Gary Illyes said noimageindex blocks a page's images and, he added, its videos too, because a video's thumbnail is an image (not in Google's docs). Author’s view: a noindex in the served HTML must never be one that JavaScript is expected to lift, and noimageindex does not belong on video watch pages.

Based on D2-C054, D2-C055, D2-C057, D2-C058, D2-C020, D2-C022, D2-C069, D2-C059, D2-C061, D2-C068, D2-C063, D2-C064, D2-C099, D2-C100, D2-C697, D2-C698, D1-C106, D2-C106, D2-C108, D2-C107, D2-C051, D2-C052, D3-C631, D3-C632, D3-C639, D2-C850, D2-C852, D2-C853, D2-C854, D2-C821, D2-C934, D2-C935, D2-C855, D2-C945

46 claims · raised in 7 sessions · said or shown on Day 2 and Day 3

Open in Reef mapOpen in Graph

Things in this topic 15

3 more things

Counts are claims that name the thing. All things

What to do

  • To keep a page out of Search, let Google crawl it and serve noindex; never combine noindex with a robots.txt disallow.
  • Ship robots rules in the HTML the server sends and never rely on JavaScript to remove a noindex.
  • Audit robots tags set by templates and plug-ins: drop an explicit all, and replace page-level nofollow with rel=nofollow on the specific links.
  • Use unavailable_after on pages with a known end date, such as events, time-limited offers or job ads.
  • Make sure the canonical page of a group carries no noindex.
  • To hide a URL urgently, use Search Console's Removals tool as well as a noindex or 404.

Day 2: Indexing 42

Shown on screen 11

SlideD2-C019

An audience member noted that a robots.txt block does not mean noindex and that explaining this to developers is a constant struggle, and asked why Google does not treat a robots.txt block as a noindex.

From the audienceIn Day 2, 10:15 · Welcome to indexing day!Evidence slide photo, transcript

Things
  • Answered by D2-C020 Day 2: Google answered on a Q&A slide that it does not treat a robots.txt disallow as a noindex because some…
  • Answered by D2-C021 Day 2: Google's Q&A slide said Google does not index the content of pages disallowed in robots.txt, but users will…
  • Answered by D2-C844 Day 2: Google illustrated why it does not treat a robots.txt disallow as noindex with an extremely important site…
  • Answered by D2-C845 Day 2: Google said very few URLs disallowed by robots.txt are in its index, compared with the index as a whole (no…
  • Answered by D2-C846 Day 2: Google said a URL disallowed by robots.txt might still be indexed if the URL is important, in which case the…
SlideConsistent with docsD2-C053

John Mueller opened with a true-or-false quiz slide asking whether robots meta tags can make a page more visible in Search results than having none, and later answered that the statement is true.

“You can use robots meta tags to be more visible in Search results than without robots meta tags.”

Wording checked against the slide or recording

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo, transcript

Things

Used byrequirement DEV-IDX-04

SlideConfirmed by docsD2-C057

The robots meta tag is written as <meta name="robots" content="rule1,rule2">, or with the name googlebot instead of robots, with several rules separated by commas.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo

Things

Used byglossary term Robots meta tag

SlideConfirmed by docsD2-C059

The robots meta rule all is the default value and does nothing: it places no restrictions on indexing the page or following its links, the same as having no robots meta tag, and it is not an instruction that search engines must index the page.

“This is the default value - it does nothing”

Wording checked against the slide or recording

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo, transcript

Things

Used byrequirement DEV-IDX-06

SlideNot in docsD2-C102

John Mueller's slide said Search Console has very few indexing-like settings and listed the Search generative AI control as the one control in his talk that is not a meta tag.

“Very few "indexing-like" settings are in Search Console”

Wording checked against the slide or recording

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo, transcript

Things
SlideConsistent with docsD2-C106

A slide said robots meta rules can be added to a page's HTML with JavaScript, but doing so takes more time.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo, transcript

Things

Used byrequirement DEV-REN-05

  • Extended by D2-C853 Day 2: John Mueller recommended putting robots meta tags directly in a page's HTML exactly as intended, and adding…
SlideConsistent with docsD2-C108

Removing a robots restriction such as noindex with JavaScript does not work, a slide said.

“But... it takes more time, and removing restrictions (like "noindex") doesn't work.”

Wording checked against the slide or recording

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence slide photo, transcript

Things

Used byrequirement DEV-REN-05

  • Extended by D2-C852 Day 2: When Google finds a noindex rule in a page's HTML, it drops the page without even processing its JavaScript…

Said on stage 18

StageConsistent with docsD2-C049

The robots meta element is also extracted when Google processes a page's HTML, and the speaker called it probably one of the most important extracted elements, or one the audience is probably interested in.

Speaker Cherry PrommawinIn Day 2, 10:25 · How is HTML interpretedEvidence transcript

StageD2-C052

John Mueller encouraged attendees to help write a specification for robots meta tags, saying internet standards groups are made up of ordinary people and need only passion and a willingness to join the discussions.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things
  • Repeated by D2-C217 Day 2: Natalia Venditto opened by saying that Gary had said earlier at the event that everyone can write a standard…
StageConfirmed by docsD2-C058

John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byglossary term robots.txt

  • Extends D1-C110 Day 1: A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google…
StageConsistent with docsD2-C063

The page-level nofollow robots rule tells search engines not to pass signals to any of the links on the page, which John Mueller called a weird and very broad rule that makes the page stand on its own.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-IDX-06glossary term nofollow

  • Extended by D2-C821 Day 2: John Mueller suspected that if robots meta tags were reinvented today, the page-level nofollow rule would…
StageConsistent with docsD2-C064

John Mueller recommends rel=nofollow on individual links instead of the page-level nofollow robots rule, so a site can choose which links are useful and which are not.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-IDX-06

StageConfirmed by docsD2-C068

The robots rule none is equivalent to noindex plus nofollow, so a page that carries it will not show up in Search, provided Google can see the tag.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-IDX-06

StageConfirmed by docsD2-C099

The noimageindex rule tells Google not to index any of the images on the page, and John Mueller said he could not see why a site would want that.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Used byrequirement DEV-IDX-09

  • Repeated by D2-C934 Day 2: The noimageindex robots meta tag tells Google not to index the images on the page.
StageConfirmed by docsD2-C100

The unavailable_after rule lets a page drop out of search results after a set date and time, which suits time-bound pages, though John Mueller said most sites do not use it.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Used byrequirement DEV-IDX-08glossary term unavailable_after

  • Extended by D2-C698 Day 2: Index selection checks for the robots unavailable_after rule and its date, and drops a page from the index…
StageConfirmed by docsD2-C850

John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-IDX-01

  • Repeats D1-C110 Day 1: A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google…
StageConsistent with docsD2-C852

When Google finds a noindex rule in a page's HTML, it drops the page without even processing its JavaScript, so a script cannot switch the page back to indexable, John Mueller said.

“we will see the noindex and say, oh, we will get rid of this page; we won't even process the JavaScript”

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-REN-05

  • Extends D2-C108 Day 2: Removing a robots restriction such as noindex with JavaScript does not work, a slide said.
StageConfirmed by docsD2-C853

John Mueller recommended putting robots meta tags directly in a page's HTML exactly as intended, and adding them with JavaScript only where that is not possible, as in a JavaScript web app.

Speaker John MuellerIn Day 2, 10:30 · Controlling indexingEvidence transcript

Things

Used byrequirement DEV-REN-05

  • Extends D2-C106 Day 2: A slide said robots meta rules can be added to a page's HTML with JavaScript, but doing so takes more time.
StageConsistent with docsD2-C432

The canonical leader of a group should be indexable: it should carry no noindex robots directive, return no error status code and not be blocked in robots.txt.

Speaker Tobias SchwarzIn Day 2, 12:05 · Lightning session E: Managing Duplicates and Site MovesEvidence transcript

Things

Used byrequirement DEV-CAN-04

StageConfirmed by docsD2-C934

The noimageindex robots meta tag tells Google not to index the images on the page.

Speaker Gary IllyesIn Day 2, 13:50 · Using images to your advantage and Engaging Search users with videosEvidence transcript

Things

Used byrequirement DEV-IDX-09glossary term noimageindex

  • Repeats D2-C099 Day 2: The noimageindex rule tells Google not to index any of the images on the page, and John Mueller said he could…
StageConsistent with docsD2-C697

Index selection drops a document carrying a noindex rule if it was not already dropped earlier in processing (noindex is a likely but not certain reading of the transcript, supported by the later mention of noindex among the Page indexing report reasons).

Speaker GoogleIn Day 2, 15:40 · Deciding what goes in the index?Evidence transcript

Things
  • Extends D1-C106 Day 1: The noindex rule consumes crawl budget, because Google must fetch the page to see it.
StageConsistent with docsD2-C698

Index selection checks for the robots unavailable_after rule and its date, and drops a page from the index once that date is reached.

Speaker GoogleIn Day 2, 15:40 · Deciding what goes in the index?Evidence transcript

Things

Used byrequirement DEV-IDX-08glossary term unavailable_after

  • Extends D2-C100 Day 2: The unavailable_after rule lets a page drop out of search results after a set date and time, which suits…

What Google's documentation says 7

DocsSourceD2-C022

Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.

Publisher Google Search CentralAnnotates Day 2, 10:15 · Welcome to indexing day!

Things

Used byrequirement DEV-IDX-01glossary term noindex

  • Extends D1-C106 Day 1: The noindex rule consumes crawl budget, because Google must fetch the page to see it.
DocsSourceD2-C831

Google's page on valid page metadata says that once Google detects an invalid element in the head, it assumes the head has ended and stops reading further elements there; only title, meta, link, script, style, base, noscript and template elements belong in the head.

Publisher Google Search CentralAnnotates Day 2, 10:25 · How is HTML interpreted

Used byrequirements DEV-CAN-03, DEV-HTM-05

DocsSourceD2-C069

Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.

Publisher Google Search CentralAnnotates Day 2, 10:30 · Controlling indexing

Things

Used byrequirements DEV-IDX-01, DEV-IDX-03glossary term X-Robots-Tag

  • Extends D1-C106 Day 1: The noindex rule consumes crawl budget, because Google must fetch the page to see it.
DocsSourceD2-C854

Google's JavaScript SEO basics guide says that when Google encounters a noindex rule it may skip rendering and JavaScript execution, so using JavaScript to change or remove a noindex robots meta tag may not work as expected.

“it may skip rendering and JavaScript execution”

Publisher Google Search CentralAnnotates Day 2, 10:30 · Controlling indexing

Things

Used byrequirement DEV-REN-05

Analysis by the author 6

AnalysisD2-C109

Ship robots rules in the HTML the server sends and never rely on JavaScript to lift a noindex: Google may skip rendering a page that arrives with noindex, so the page can stay out of the index even if a script removes the tag later.

Author Ibrahim AnjroAnnotates Day 2, 10:30 · Controlling indexing

Things

Used byrequirement DEV-REN-05

  • Extended by D2-C855 Day 2: On stage the rule was absolute (Google won't even process the JavaScript of a page served with noindex)…
AnalysisD2-C855

On stage the rule was absolute (Google won't even process the JavaScript of a page served with noindex), while Google's guide says only that it may skip rendering; either way, a noindex in the served HTML must never be one that JavaScript is expected to lift.

Author Ibrahim AnjroAnnotates Day 2, 10:30 · Controlling indexing

Things
  • Extends D2-C109 Day 2: Ship robots rules in the HTML the server sends and never rely on JavaScript to lift a noindex: Google may…
AnalysisD2-C945

Do not set noimageindex on video watch pages: Gary Illyes said it also stops the video, through its thumbnail, and although the robots meta tag specification mentions only images, the Video indexing report treats a missing or blocked thumbnail as a reason a video is not indexed.

Author Ibrahim AnjroAnnotates Day 2, 13:50 · Using images to your advantage and Engaging Search users with videos

Things
AnalysisD2-C699

Use the unavailable_after robots rule on pages with a known end date, such as event pages, time-limited offers or job ads, so that index selection drops them automatically when the date passes instead of leaving expired pages in search results.

Author Ibrahim AnjroAnnotates Day 2, 15:40 · Deciding what goes in the index?

Things

Used byrequirement DEV-IDX-08

Day 3: Serving: Ranking, Search Console, and Performance 4

Shown on screen 1

Said on stage 2

Analysis by the author 1

Across days and sessions 12

  1. Docs D2-C022 Day 2 · Welcome to indexing day!

    Google's noindex documentation says the noindex rule works only if the page is not blocked by robots.txt: a crawler that cannot fetch the page never sees the rule, and the page can still appear in search results, for example if other pages link to it.

    extends
    Slide D1-C106 Day 1 · How Google thinks about crawl budget

    The noindex rule consumes crawl budget, because Google must fetch the page to see it.

  2. Stage D2-C058 Day 2 · Controlling indexing

    John Mueller said robots.txt does not control indexing, so robots meta tags are what site owners have to use to control it.

    extends
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

  3. Docs D2-C069 Day 2 · Controlling indexing

    Google's robots meta tag specification says robots meta tags and X-Robots-Tag headers are found only when a URL is crawled, so the rules on a URL disallowed in robots.txt are never seen and are ignored.

    extends
    Slide D1-C106 Day 1 · How Google thinks about crawl budget

    The noindex rule consumes crawl budget, because Google must fetch the page to see it.

  4. Stage D2-C697 Day 2 · Deciding what goes in the index?

    Index selection drops a document carrying a noindex rule if it was not already dropped earlier in processing (noindex is a likely but not certain reading of the transcript, supported by the later mention of noindex among the Page indexing report reasons).

    extends
    Slide D1-C106 Day 1 · How Google thinks about crawl budget

    The noindex rule consumes crawl budget, because Google must fetch the page to see it.

  5. Stage D2-C698 Day 2 · Deciding what goes in the index?

    Index selection checks for the robots unavailable_after rule and its date, and drops a page from the index once that date is reached.

    extends
    Stage D2-C100 Day 2 · Controlling indexing

    The unavailable_after rule lets a page drop out of search results after a set date and time, which suits time-bound pages, though John Mueller said most sites do not use it.

  6. Stage D2-C821 Day 2 · Controlling indexing

    John Mueller suspected that if robots meta tags were reinvented today, the page-level nofollow rule would probably not be part of them.

    extends
    Stage D2-C063 Day 2 · Controlling indexing

    The page-level nofollow robots rule tells search engines not to pass signals to any of the links on the page, which John Mueller called a weird and very broad rule that makes the page stand on its own.

  7. Stage D2-C852 Day 2 · Controlling indexing

    When Google finds a noindex rule in a page's HTML, it drops the page without even processing its JavaScript, so a script cannot switch the page back to indexable, John Mueller said.

    extends
    Slide D2-C108 Day 2 · Controlling indexing

    Removing a robots restriction such as noindex with JavaScript does not work, a slide said.

  8. Stage D2-C853 Day 2 · Controlling indexing

    John Mueller recommended putting robots meta tags directly in a page's HTML exactly as intended, and adding them with JavaScript only where that is not possible, as in a JavaScript web app.

    extends
    Slide D2-C106 Day 2 · Controlling indexing

    A slide said robots meta rules can be added to a page's HTML with JavaScript, but doing so takes more time.

  9. Analysis D2-C855 Day 2 · Controlling indexing

    On stage the rule was absolute (Google won't even process the JavaScript of a page served with noindex), while Google's guide says only that it may skip rendering; either way, a noindex in the served HTML must never be one that JavaScript is expected to lift.

    extends
    Analysis D2-C109 Day 2 · Controlling indexing

    Ship robots rules in the HTML the server sends and never rely on JavaScript to lift a noindex: Google may skip rendering a page that arrives with noindex, so the page can stay out of the index even if a script removes the tag later.

  10. Stage D2-C217 Day 2 · Lightning session D: Rendering and JavaScript

    Natalia Venditto opened by saying that Gary had said earlier at the event that everyone can write a standard, and that this is what the Web Fragments work is trying to do.

    repeats
    Stage D2-C052 Day 2 · Controlling indexing

    John Mueller encouraged attendees to help write a specification for robots meta tags, saying internet standards groups are made up of ordinary people and need only passion and a willingness to join the discussions.

  11. Stage D2-C850 Day 2 · Controlling indexing

    John Mueller said robots meta rules only work if robots.txt allows Google to fetch the page.

    repeats
    Analysis D1-C110 Day 1 · How Google thinks about crawl budget

    A URL blocked in robots.txt can still be indexed without its content if other pages link to it, and Google cannot see a noindex on a page it is not allowed to fetch.

  12. Stage D2-C934 Day 2 · Using images to your advantage and Engaging Search users with videos

    The noimageindex robots meta tag tells Google not to index the images on the page.

    repeats
    Stage D2-C099 Day 2 · Controlling indexing

    The noimageindex rule tells Google not to index any of the images on the page, and John Mueller said he could not see why a site would want that.

Built on these claims 12

Developer requirements 12

Sources 16