The control does not affect AI training. Training of the models is limited with Google-Extended instead.
Google Search Console Help · Day 1 · What's new in the world of Search
Thing · Crawler
Google-Extended
A robots.txt control token, not a separate crawler, that decides whether crawled content may be used for Gemini models and grounding outside Search.
- Claims
- 12
- In Google’s docs
- 3
- Said at the event
- 4
- Not in docs
- 1
- Kit items
- 4
Glossary · Google-Extended
A robots.txt control token, not a separate crawler, that decides whether crawled content may train future Gemini models and ground Gemini apps and Vertex AI. It does not affect inclusion or ranking in Google Search. A site can disallow it for a directory or for the whole site.
Google’s documentation 3
Documented in
Google-Extended is a control token, not a crawler with its own user agent string. It decides whether crawled content may be used to train future Gemini models and to ground Gemini apps and Vertex AI.
Google's crawler documentation defines grounding in Gemini Apps and in Grounding with Google Search on Vertex AI as providing content from the Google Search index to the model at prompt time, and sites manage whether their content is used for it with the Google-Extended robots.txt token.
Google · Day 2 · Lightning session D: Rendering and JavaScript
Said at the event 4
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Not in docs 1
A Google panelist said Apple offers a robots.txt control for Applebot similar to Google-Extended, so a site can allow Applebot for search while opting out of Apple's AI use.
Consistent with docs 1
Google said a site can keep a directory or the whole site out of Gemini model training by disallowing it for the Google-Extended token in robots.txt, and that Google respects that policy; the speaker believed Google was the first to offer such an opt-out from model training.
Confirmed by docs 1
To opt out of Google's AI training, a site can use the Google-Extended token in robots.txt.
Nothing to verify 1
The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.
Press and analysis 5
AnalysisD1-C523
The 'first' needs care: OpenAI documented how to block its GPTBot crawler in robots.txt in August 2023, before Google announced Google-Extended on 28 September 2023. What Google-Extended added was a token that controls AI-training use of pages fetched by Google's existing crawlers, separately from Search; Apple later introduced a similar token, Applebot-Extended.
Ibrahim Anjro · Day 1 · How Google interprets robots.txt
AnalysisD1-C486
The panel spoke of blocking Google's 'AI crawling or training' with Google-Extended, but Google documents Google-Extended as a usage token, not a crawler: disallowing it does not stop Googlebot fetching pages and only controls use for Gemini training and grounding.
Ibrahim Anjro · Day 1 · Q&A
AnalysisD2-C067
nofollow is a link-graph hint for search engines, not an access control: to keep AI crawlers away from content, use robots.txt rules for the specific crawlers; at Google, Google-Extended covers Gemini training and grounding, and the Search generative AI control covers AI Overviews and AI Mode.
Ibrahim Anjro · Day 2 · Controlling indexing
AnalysisD2-C119
If Gemini training reuses the rendering done for Search, a site needs no separate rendering work for Gemini; whether its rendered content is used for training is decided with the Google-Extended token in robots.txt, not by rendering choices.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
AnalysisD2-C123
Google documents Gemini grounding only as content from the Search index at prompt time; the live read of a specific page at a user's request, described on stage, is not documented, so it is unclear whether it works like a user-triggered fetcher, which generally ignores robots.txt, or follows Google-Extended.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
Built on these claims 4
Kit items about Google-Extended: their own words name it, or several of the claims they rest on do.
Developer requirements 2
Use the Google-Extended robots.txt token to control Gemini training and grounding
Rests on 8 claims, 6 of them naming Google-Extended; its own words name Google-Extended
Rests on 13 claims, 3 of them naming Google-Extended; its own words name Google-Extended
Also inglossary terms Google-Extended, Grounding
Connected things 10
Relations
- Affects Gemini
3 claims, 3 documented
Google-Extended is a control token, not a crawler with its own user agent string. It decides whether crawled content may be used to train future Gemini models and to ground Gemini apps and Vertex AI.
Google said a site can keep a directory or the whole site out of Gemini model training by disallowing it for the Google-Extended token in robots.txt, and that Google respects that policy; the speaker believed Google was the first to offer such an opt-out from model training.
Google's crawler documentation defines grounding in Gemini Apps and in Grounding with Google Search on Vertex AI as providing content from the Google Search index to the model at prompt time, and sites manage whether their content is used for it with the Google-Extended robots.txt token.
Most often named with it
Things named in the same claim, with the number of claims they share.
- robots.txt 9
- Gemini 7
- Grounding 4
- Googlebot 2
- AI Mode 1
- AI Overviews 1
- nofollow 1
- Search generative AI control 1
- User-triggered fetchers 1