Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Thing · Crawler
Google crawlers for specific products under an agreement with the site (such as AdsBot), which may ignore the global robots.txt user agent.
Glossary · Contractual crawlers (special-case crawlers)
Google crawlers that serve a specific product by agreement with the crawled site, and the only Google crawlers that may ignore robots.txt; AdsBot, for example, ignores the global user agent with the ad publisher's permission.
Documented in
Google's crawler documentation says special-case crawlers serve specific Google products where the crawled site and the product have an agreement about the crawl process, so they may ignore robots.txt rules; AdsBot, for example, ignores the global (*) user agent with the ad publisher's permission.
Google · Day 1 · How crawling works
Google's page on verifying its crawlers says Googlebot and Google's other common crawlers resolve to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com host names, special-case crawlers to rate-limited-proxy-*.google.com and user-triggered fetchers to *.gae.googleusercontent.com or google-proxy-*.google.com, and it publishes each group's IP ranges as JSON files such as common-crawlers.json and special-crawlers.json.
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Apart from contractual crawlers, all of Google's automated crawlers obey robots.txt, which Google treats as the way site owners opt out of its crawling.
Gary Illyes · Day 1 · How crawling works
The only Google-owned crawlers that do not obey robots.txt are contractual crawlers, which crawl a site whose owner has agreed that Google may crawl it however it likes.
Gary Illyes · Day 1 · How crawling works
Kit items about Special-case crawlers: their own words name it, or several of the claims they rest on do.
Also inglossary term Contractual crawlers (special-case crawlers)
Things named in the same claim, with the number of claims they share.