Google's page on verifying its crawlers says Googlebot and Google's other common crawlers resolve to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com host names, special-case crawlers to rate-limited-proxy-*.google.com and user-triggered fetchers to *.gae.googleusercontent.com or google-proxy-*.google.com, and it publishes each group's IP ranges as JSON files such as common-crawlers.json and special-crawlers.json.
Thing · Crawler
User-triggered fetchers
Google fetchers that request a page because a user asked for it (for example a site verification), and that generally ignore robots.txt.
- Claims
- 4
- In Google’s docs
- 2
- Said at the event
- 1
- Not in docs
- 0
- Kit items
- 3
Glossary · User-triggered fetchers
Fetchers that request a page because a user asked for it, such as a translation service. Google's documentation says they generally ignore robots.txt, and Google said AI agents acting for a user fall into the same category.
Google’s documentation 2
Documented in
Google says user-triggered fetchers, which fetch a URL because a user asked for it in a Google product (for example Gemini Notebook fetching URLs users add as sources, or Google-Agent acting on a user's request), generally ignore robots.txt rules.
Google · Day 2 · Lightning session D: Rendering and JavaScript
Said at the event 1
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Confirmed by docs 1
A community speaker pointed out that, according to Google's documentation, user-triggered fetchers, which fetch a page because a user asked for it, generally ignore robots.txt rules.
Carlos Ortega · Day 1 · Lightning session A: Automation and AI
Press and analysis 1
AnalysisD2-C123
Google documents Gemini grounding only as content from the Search index at prompt time; the live read of a specific page at a user's request, described on stage, is not documented, so it is unclear whether it works like a user-triggered fetcher, which generally ignores robots.txt, or follows Google-Extended.
Ibrahim Anjro · Day 2 · Lightning session D: Rendering and JavaScript
Built on these claims 3
Kit items about User-triggered fetchers: their own words name it, or several of the claims they rest on do.
Developer requirements 2
Rests on 13 claims, 1 of them naming User-triggered fetchers; its own words name User-triggered fetchers
Use the Google-Extended robots.txt token to control Gemini training and grounding
Rests on 8 claims, 1 of them naming User-triggered fetchers; its own words name User-triggered fetchers
Also inglossary term User-triggered fetchers
Connected things 6
Most often named with it
Things named in the same claim, with the number of claims they share.