Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Thing · Crawler
Google Search's main crawler, which fetches pages for the index as a smartphone or desktop user agent.
Documented in
Google's Inside Googlebot post (March 2026) says Googlebot currently fetches only the first 2MB of each URL, HTTP headers included (64MB for PDFs); bytes past that cutoff are not fetched, rendered or indexed, and each resource the page loads has its own separate limit.
Search Central blog (31 March 2026) · Day 1 · How crawling works
Google's Inside Googlebot post warns that bloated inline base64 images, large blocks of inline CSS or JavaScript, or megabytes of menus can push a page's text or structured data past Googlebot's 2MB cutoff, and advises moving heavy CSS and JavaScript to external files and placing meta tags, the title, the canonical and essential structured data high in the HTML.
Search Central blog (31 March 2026) · Day 1 · How crawling works
Google's Inside Googlebot post (March 2026) says Googlebot is today just one user of a centralized crawling platform, and that dozens of other clients, such as Google Shopping and AdSense, send their crawl requests through the same infrastructure under other crawler names, with only the larger ones documented.
Search Central blog (31 March 2026) · Day 1 · How crawling works
Google's page on verifying its crawlers says Googlebot and Google's other common crawlers resolve to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com host names, special-case crawlers to rate-limited-proxy-*.google.com and user-triggered fetchers to *.gae.googleusercontent.com or google-proxy-*.google.com, and it publishes each group's IP ranges as JSON files such as common-crawlers.json and special-crawlers.json.
Google's guide to A/B testing says never to show one set of URLs to Googlebot and a different set to humans: that is cloaking, which is against Google's spam policies whether or not a test is running.
Google Search Central · Day 1 · How Google thinks about crawl budget
Google's Search documentation gives notranslate as a rule for a robots or googlebot meta tag or an X-Robots-Tag header, and says the rule opts a page out of all translation features in Google Search.
Google Search Central · Day 2 · Controlling indexing
Google's robots meta tag specification says Googlebot considerably decreases the crawl rate of a URL after the date and time set in its unavailable_after rule.
Google Search Central · Day 2 · Controlling indexing
Google's JavaScript SEO guide says Googlebot sends every page with a 200 HTTP status code to the rendering queue, whether or not it contains JavaScript, unless a robots meta tag or header tells Google not to index it, and Google uses the rendered HTML to index the page.
Google Search Central · Day 2 · Lightning session D: Rendering and JavaScript
Google says Googlebot and its Web Rendering Service identify resources that do not contribute to essential page content, such as reporting and error requests, and may not fetch them, so client-side analytics may not give a full or accurate picture of their activity.
Google Search Central · Day 2 · Lightning session D: Rendering and JavaScript
Google's JavaScript SEO basics guide says Googlebot extracts links twice, from the HTML response before rendering and again from the rendered HTML, so links injected with JavaScript can be found if they use crawlable <a href> markup.
Google Search Central · Day 2 · Lightning session D: Rendering and JavaScript
Google says its Web Rendering Service fetches the resources a page references through Googlebot, including JavaScript, CSS and XHR requests to APIs, but not images or videos.
Search Central blog (3 December 2024), Search Central blog (31 March 2026) · Day 2 · Lightning session D: Rendering and JavaScript
For effects that need WebGL, which Googlebot does not support, Google's guide to fixing JavaScript problems suggests skipping the effect or prerendering it with server-side rendering so that the content is accessible to Googlebot.
Google Search Central · Day 2 · What is Google friendly JavaScript
Google's documentation says Googlebot mostly crawls from US IP addresses and sends no Accept-Language header, so pages that change content or redirect by the visitor's perceived country or language may not have every version crawled, indexed or ranked; it recommends separate URLs annotated with hreflang.
Google Search Central · Day 2 · Focusing on Internationalisation and Localisation
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Bing closes the robots.txt group at an unknown line placed between two user-agent lines, so in the slide's example bingbot ends up not blocked while Googlebot is; the slide warned that different crawlers behave differently.
Dave Smart said this applies to all redirects, not only JavaScript ones; his examples: a redirect through an external authorisation service that is blocked by its own robots.txt, content that moved through several URLs over the years with one of them later blocked, and unexpected redirects, such as one served only to Googlebot's user agent.
Dave Smart · Day 1 · Lightning session B: Robots.txt
To check that new content is crawled fast enough, Gary Illyes advised measuring the time between publishing a URL and Googlebot's first crawl of it: ideally it stays flat or falls, and only a rising trend calls for investigation.
Gary Illyes · Day 1 · Q&A
Gary Illyes said that if the time from publishing to Googlebot's first crawl starts rising, how much it rises matters: for the crawling of fresh stories, something like two hours is probably not great (a smaller example figure is unclear in the recording).
Gary Illyes · Day 1 · Q&A
A community speaker compared Googlebot to an inspector who cannot enter the shop and only looks through the window, seeing the page as ones and zeros, which the speaker linked to tokenization.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
Googlebot is an ordinary HTTP client with nothing special about it: like a browser, it fetches a URL it was given and returns the fetched bytes to Google's servers.
Gary Illyes · Day 1 · How crawling works
Googlebot is the crawler Google uses for web search, including Search's AI features.
Gary Illyes · Day 1 · How crawling works
Google treats a 402 Payment Required response as a 404 Not Found, because Googlebot cannot pay for content.
Gary Illyes · Day 1 · How crawling errors affect Search
CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.
Gary Illyes · Day 1 · How crawling errors affect Search
A user-agent line with its allow and disallow rules forms a user-agent group, and one group can name several crawlers (the talk's example: Googlebot and Bingbot); give named crawlers their own group only when they need rules the * group should not grant to every crawler.
In the talk's quiz, the answer for letting Googlebot crawl /staging/preview/ while keeping the rest of /staging/ closed to it was a user-agent: googlebot group with both rules, disallow /staging/ and allow /staging/preview/, rather than an allow in the * group or a Googlebot group with only the allow.
To judge crawl efficiency, check Search Console's Crawl Stats report: whether total crawling has plateaued over time, and whether the average response time shows the server is fast enough or is limiting Googlebot.
Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in robots.txt shifts crawl budget to the rest of the site.
Gary Illyes · Day 1 · Q&A
Files such as cats.txt have no importance for Google Search: if a site links to one, Googlebot will find and crawl it, but it has no other effect.
To check whether Google lowered a site's crawl capacity limit, look at the number of connections Googlebot opens to the site: if it has dropped, the capacity limit changed.
A Google panelist called pagination one of the trickiest things in web development and said switching to infinite scroll or a load-more button is risky, depending on what you want to achieve, because Googlebot does not click buttons.
A Google panelist illustrated a crawl capacity limit drop as a step down in the site's hostload, for example from 10 to 5, which the panelist said means Googlebot then makes up to five requests per second (the recording adds 'per connection', which is unclear).
Googlebot does not support HTTP/3 today.
Google cannot extract a link from an a element that only has an onclick handler, because Googlebot does not click, so the JavaScript is never triggered.
Lazy loading triggered by a scroll event listener, such as window.addEventListener('scroll', loadMoreProducts), never runs for Googlebot because Googlebot does not scroll.
Rebecca Yu · Day 2 · Lightning session D: Rendering and JavaScript
Google named three causes of content missing from the rendered HTML: JavaScript inaccessible to Googlebot, JavaScript DOM event triggers and disabled browser APIs.
Erin Sparling · Day 2 · What is Google friendly JavaScript
When the JavaScript is not accessible to Googlebot, Google cannot render the client-side DOM that the page builds asynchronously, so parts of the page may be present while the main content is absent.
Erin Sparling · Day 2 · What is Google friendly JavaScript
A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.
Google's systems find it very hard to recognise a bot-challenge page as an error page, and Googlebot cannot solve the puzzles, such as selecting all the cats, that those pages ask for.
Google's long-standing advice to write for people and give them what they want has become true in practice because Googlebot has become more and more human, a community speaker argued.
Google Shopping uses its own crawler, Storebot (Storebot-Google), instead of Googlebot because it needs fresh product information, prices, availability and shipping details and therefore crawls much more often.
Alex Jansen · Day 3 · Shopping on Search: Beyond the blue links
A crawler is software that downloads pages, extracts their links and repeats the process on the links it extracted; Googlebot is the main crawler of Google Search.
Cherry Prommawin · Day 1 · How Search works and where's AI?
Googlebot ignores an unknown line, such as a Content-signal line, placed between two user-agent lines and joins both user agents into one group, so the rules that follow apply to both; in the slide's example both are blocked.
Dave Smart said robots.txt groups are not additive: if a group matches a crawler's user agent, only that group's rules apply, so a googlebot group that blocks /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks.
Dave Smart · Day 1 · Lightning session B: Robots.txt
The non-standard crawl-delay rule is not processed by Googlebot, so it does not save any crawl budget.
Googlebot as a single standalone crawler is a historical idea: Google crawls through a centralised crawling infrastructure, so a request from a Google user agent in server logs is a request routed through that shared platform.
If paginated pages are replaced by a load-more button or infinite scroll without crawlable links to them, Google will not see the further pages at all.
The robots meta tag is written as <meta name="robots" content="rule1,rule2">, or with the name googlebot instead of robots, with several rules separated by commas.
John Mueller · Day 2 · Controlling indexing
Storebot-Google also does deep crawls, for example of checkout pages, which do not belong in the search index; this is another reason it is kept separate from Googlebot.
Alex Jansen · Day 3 · Shopping on Search: Beyond the blue links
The robots.txt session worked through an example file with three groups: a default group that disallows everything except the bare homepage, a shared bingbot and googlebot group that also allows /politics/ and /*/live/, and a google-extended group that disallows /subscriptions/.
An audience member asked, in questions submitted before the event, how often Googlebot should be expected to visit a site now that AI Mode has rolled out, and whether AI means more crawling.
From the audience · Day 1 · How Google thinks about crawl budget
An audience member asked whether Google plans dedicated user agents for AI, as OpenAI has several, so that site owners can control AI access separately from Googlebot and analyse it in their logs.
From the audience · Day 1 · Q&A
A Google panelist called blocking all AI crawlers while allowing search crawlers such as Googlebot, Bingbot and Applebot a personal, philosophical decision that every site owner can make.
An audience member asked whether HTML sitemaps still matter for large sites, whether they improve Googlebot's crawl discovery and whether they pass link equity to deeper pages.
From the audience · Day 2 · Welcome to indexing day!
An audience member asked how several sloppy migrations on the same domain can affect Googlebot's crawling.
From the audience · Day 2 · Welcome to indexing day!
In Lightning session K (Facets of quality), a community speaker argued that SEO is for humans: write for people and the problems they have, rather than to please Googlebot.
Because Googlebot is not a human, the speaker's team at the time thought more about Googlebot than about human readers, a community speaker said.
On stage Google spoke of probably hundreds, if not thousands, of crawlers, while its Inside Googlebot post speaks of dozens of other clients; both agree that only the larger crawlers are documented, so a Google user agent missing from the public lists is not proof of a fake request, and reverse DNS or Google's published IP ranges are the test.
Ibrahim Anjro · Day 1 · How crawling works
For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.
Ibrahim Anjro · Day 1 · How crawling errors affect Search
Do not answer verified Googlebot with 401, 402 or 403, for example from a login wall, a paywall or a pay-per-crawl setup: Google treats them like 404 and drops the pages, and Google's status code page says not to use 401 or 403 to limit crawling; to slow crawling temporarily, return 429 or 503.
Ibrahim Anjro · Day 1 · How crawling errors affect Search
Under the example file, Googlebot may crawl /, /politics/eu-vote and /sports/live/, and is blocked from /?utm_source=x, /index.html, /politics (no trailing slash), /live/ and /sports/live-score. An 'allow: /$' rule does not cover the homepage with tracking parameters.
Ibrahim Anjro · Day 1 · How Google interprets robots.txt
A 404 fetch is still a fetch: Google's 2017 crawl budget post says generally any URL Googlebot crawls counts towards a site's crawl budget, so the stage point that 4xx responses do not affect crawl budget is best read as 'they do not slow crawling, and a 404 tells Google to crawl that URL less over time'.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
The panel spoke of blocking Google's 'AI crawling or training' with Google-Extended, but Google documents Google-Extended as a usage token, not a crawler: disallowing it does not stop Googlebot fetching pages and only controls use for Gemini training and grounding.
Ibrahim Anjro · Day 1 · Q&A
Google's crawl budget guide does not express the crawl capacity limit in requests per second: it limits the total time a server spends holding connections open for Google, counting both the number of parallel connections and their duration. That fits the panel's advice to watch how many connections Googlebot opens (D1-C494): fewer connections is the documented form of a lower capacity limit.
Ibrahim Anjro · Day 1 · Q&A
International sites should remove notranslate from their templates unless translation must be prevented: the robots or googlebot form switches off Google Search's translation features, and John Mueller said the form named google also blocks Chrome's translation for visitors who do not read the language.
Ibrahim Anjro · Day 2 · Controlling indexing
Check what your CDN or bot protection serves to verified Googlebot and to the AI agents you want to allow; if crawlers must get a challenge page, return it with a 503 status as Google recommends, never as a 200 page across many URLs, which Google may cluster as duplicates.
Ibrahim Anjro · Day 2 · Handling web duplication
The inspector metaphor blends two steps: Googlebot fetches pages during crawling, while tokenization happens later when the page is processed for the index, as Google explained on Day 2.
Ibrahim Anjro · Day 3 · Lightning session K: Facets of quality
For developers, Google's five listed spam types map to concrete checks: serve Googlebot the same content as users (cloaking), avoid near-identical location or keyword pages (doorways), add value to reused content (scraped content), and patch and monitor the CMS against injected pages and links (hacked content).
Ibrahim Anjro · Day 3 · What are quality updates
Treat Storebot-Google separately from Googlebot in robots.txt and bot protection: a shop that blocks it on product, cart or checkout paths can undermine Shopping price and availability checks, even when Googlebot is allowed.
Ibrahim Anjro · Day 3 · Shopping on Search: Beyond the blue links
Kit items about Googlebot: their own words name it, or several of the claims they rest on do.
Never serve bot challenges or error pages to crawlers with a 200 status
Rests on 7 claims, 5 of them naming Googlebot
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 4 of them naming Googlebot; its own words name Googlebot
Keep connect time and time to first byte low and stable under crawler load
Rests on 13 claims, 3 of them naming Googlebot
Let Storebot-Google crawl product, cart and checkout pages, through robots.txt and bot protection
Rests on 6 claims, 3 of them naming Googlebot
Let verified Google crawlers through firewalls, CDNs and bot protection
Rests on 11 claims, 3 of them naming Googlebot
Give a crawler its own robots.txt group only when needed, and make that group complete
Rests on 5 claims, 3 of them naming Googlebot; its own words name Googlebot
Rests on 8 claims, 2 of them naming Googlebot; its own words name Googlebot
Measure the time from publishing a URL to Googlebot's first crawl of it
Rests on 7 claims, 2 of them naming Googlebot; its own words name Googlebot
Keep each HTML response well under 2 MB and put the main content and JSON-LD early
Rests on 2 claims, 2 of them naming Googlebot; its own words name Googlebot
Do not put Content-Signal or other non-standard lines between user-agent lines
Rests on 6 claims, 2 of them naming Googlebot; its own words name Googlebot
Back infinite scroll and load-more buttons with paginated URLs
Rests on 6 claims, 2 of them naming Googlebot; its own words name Googlebot
Never route a crawlable URL's redirects through a URL that robots.txt blocks
Rests on 4 claims, 1 of them naming Googlebot; its own words name Googlebot
Use unavailable_after on pages with a known end date
Rests on 4 claims, 1 of them naming Googlebot; its own words name Googlebot
Give every language and country version its own URL
Rests on 4 claims, 1 of them naming Googlebot; its own words name Googlebot
Do not redirect visitors automatically by IP address or browser language
Rests on 2 claims, 1 of them naming Googlebot; its own words name Googlebot
Lazy-load with native lazy loading or an IntersectionObserver, never with scroll events
Rests on 6 claims, 1 of them naming Googlebot; its own words name Googlebot
Do not block JavaScript, CSS or API endpoints that pages need for rendering in robots.txt
Rests on 7 claims, 1 of them naming Googlebot; its own words name Googlebot
Never serve Googlebot different content from users, and do not build doorway pages
Rests on 4 claims, 1 of them naming Googlebot; its own words name Googlebot
Answer crawlers over HTTP/1.1 (HTTP/2 optional); never run a crawlable host on HTTP/3 only
Rests on 4 claims, 1 of them naming Googlebot; its own words name Googlebot
Do not answer crawlers with 401, 402 or 403 on pages that should be in Search
Rests on 5 claims, 1 of them naming Googlebot; its own words name Googlebot
Make every navigational link an <a> element whose href holds a real URL
Rests on 6 claims, 1 of them naming Googlebot; its own words name Googlebot
Have all indexable content in the DOM after load, without clicks, typing or scrolling
Rests on 8 claims, 0 of them naming Googlebot; its own words name Googlebot
Also inglossary terms User-agent group, cats.txt, Crawl rate limit (hostload), Storebot-Google
Things named in the same claim, with the number of claims they share.