Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · How Search works
Search works in three stages, crawling, indexing and serving, and AI Mode and AI Overviews share the crawling and indexing stages with classic Search, adding grounding on the index and query fan-out only at serving; Day 2 repeated that they are a different experience of the same content. Day 2 filled in the middle: a processing stage between the crawler and the index runs HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection, and sends the links it finds back to the crawl queue. Google's how-Search-works guide describes rendering as part of the crawl, while the event slides drew it inside processing. Gemini is not part of Search but uses Google's crawlers and grounds on the Search index; Day 2 qualified the shared tokenization mentioned on Day 1, showing that Gemini's tokenizer splits text differently from the one Search uses. Day 3 drew the serving stage: query understanding and retrieval lead into the index, then ranking and search features lead back to the user, and AI features send their fan-out queries back through the same search engine. Google said its processes depend on each other, crawling, then indexing, then serving, so a change such as a new canonical cannot be processed until the page is recrawled, and that AI Overviews and AI Mode are built on Search infrastructure 25 to 30 years old with very few processes of their own. Day 1's second recording added that crawling and indexing happen before anyone searches while serving and ranking happen in real time, and that AI Overviews and AI Mode may have extra processes of their own like any search feature but share the bulk of their processing with normal Search.
Things in this topic 12
Counts are claims that name the thing. All things
What to do
Knowing how Search works largely means knowing how the AI features work.
“If you know how Search works, you largely also know how AI features work.”
Wording checked against the slide or recording
Speaker Gary IllyesIn Day 1, 11:35 · What's new in the world of SearchEvidence slide photo, transcript
Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.
Speaker Cherry Prommawin, Gary IllyesIn Day 1, 11:45 · How Search works and where's AI?Evidence 2 slide photos, transcript
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
Speaker Cherry Prommawin, Gary IllyesIn Day 1, 11:45 · How Search works and where's AI?Evidence slide photo, transcript
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Speaker Cherry Prommawin, Gary IllyesIn Day 1, 11:45 · How Search works and where's AI?Evidence slide photo, transcript
Used byrequirement DEV-AIF-01story angle A-001glossary terms AI Mode, Grounding
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Speaker Cherry Prommawin, Gary IllyesIn Day 1, 11:45 · How Search works and where's AI?Evidence slide photo, transcript
AI Overviews and AI Mode may have extra processes of their own, like any other search feature, but the bulk of their processing is the same as for normal Search.
Speaker Gary IllyesIn Day 1, 11:45 · How Search works and where's AI?Evidence transcript
Crawling and indexing happen before anyone searches, while serving and ranking happen in real time when someone types a query.
Speaker Cherry PrommawinIn Day 1, 11:45 · How Search works and where's AI?Evidence transcript
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Speaker Cherry PrommawinIn Day 2, 10:25 · How is HTML interpretedEvidence slide photo
Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.
Speaker Gary IllyesIn Day 2, 13:30 · Finding the gold nuggets: structured data, media, and more!Evidence slide photo, transcript
Opening Day 2, Google recapped that Day 1 began with how Search works and then covered crawling, including basics of how the internet works such as TCP/IP, so that attendees could understand how crawlers work.
Speaker not identifiedIn Day 2, 10:15 · Welcome to indexing day!Evidence transcript
AI Overviews and AI Mode are built on top of Search results: they are a different experience of the same content Google already has.
Speaker Erin SparlingIn Day 2, 10:40 · Lightning session D: Rendering and JavaScriptEvidence transcript
Used bystory angle A-001
The speaker recapped Google's pipeline up to the index: Google crawls pages, processes the fetched documents and then stores them in its index.
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
Google's guide to how Search works says rendering happens during the crawl, and describes indexing as analysing a page's text, key tags and attributes such as title elements and alt attributes, images and videos, and deciding whether the page is a duplicate or the canonical.
Publisher Google Search CentralAnnotates Day 2, 13:30 · Finding the gold nuggets: structured data, media, and more!
Used byrequirement DEV-HTM-03
Google's opening slide for serving day showed serving as the third stage after crawling and indexing, and drew Google's serving infrastructure as query understanding and retrieval leading into the index, then ranking and search features leading back to the user, for the example query 'Where to eat jamon'.
Speaker GoogleIn Day 3, 10:15 · Welcome to serving and ranking day!Evidence slide photo
Google's slide on how LLM features with grounding generally work showed a query going to both the search engine and an LLM, the search engine's results going to the LLM, the LLM generating fan-out queries that go back to the search engine, and the LLM returning answers with links.
“How LLM features with grounding generally work”
Wording checked against the slide or recording
Speaker John MuellerIn Day 3, 10:25 · Making sense of users' queriesEvidence slide photo, transcript
Google's serving diagram shows the query passing through query understanding and retrieval to the index, then back through ranking and search features to the user, with the Search Features step highlighted (example query 'Where to eat orange').
Speaker Gary IllyesIn Day 3, 13:25 · How Search results are bornEvidence slide photo
Google placed query understanding in the traditional part of Search, the part where answers are looked up in an index and then presented and ranked.
Speaker John MuellerIn Day 3, 10:25 · Making sense of users' queriesEvidence transcript
After query understanding and expansion, Google sends the queries to the index to look up pages, the retrieval step.
Speaker John MuellerIn Day 3, 10:25 · Making sense of users' queriesEvidence transcript
Used byglossary term Retrieval
Google's processes depend on each other: crawling happens first, then indexing, then serving, so a change such as a new canonical cannot be processed until the page has been recrawled.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
Used byrequirement DEV-MON-10
Google said it structured the three days of the event as crawling, indexing and serving because that is literally how Search works.
Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript
AI Overviews and AI Mode are built on the Search infrastructure Google has used for 25 to 30 years and have very few processes of their own.
Speaker Gary IllyesIn Day 3, 16:00 · Wrapping all up: AI, Search, and making sense of everything.Evidence transcript
Used bystory angle A-001
Tokenization for AI models such as Gemini, in training and in inference, differs from tokenization for Search, although Gary Illyes qualified this with 'or mostly'.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Signals calculated for a page during indexing are stored in the index and used both to decide whether the page gets indexed and, later, for ranking.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
AI Overviews and AI Mode may have extra processes of their own, like any other search feature, but the bulk of their processing is the same as for normal Search.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Index selection runs after signals are collected and duplicates are dropped, and decides what goes into Google's index, which is big but not limitless.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
Crawling and indexing happen before anyone searches, while serving and ranking happen in real time when someone types a query.
Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.
Google said it does not own third-party AI chatbots such as ChatGPT and has no insight into how they work or are built.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Googlebot is the crawler Google uses for web search, including Search's AI features.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Crawling for Gemini may be set to care less about quality and more about the amount of content, because for large language models the number of tokens matters more than quality.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
AI Mode is a feature that sits on top of Google's existing crawling infrastructure, so Google does not crawl a site more because of AI Mode.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.
John Mueller said Google uses the snippet as a way of building AI Overviews and AI Mode answers, so if a page forbids a snippet, Google cannot use that snippet for them.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Google's AI features guide says a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, with no additional technical requirements.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
To train Gemini models, Google renders every page just as it does for Search, so a page that renders correctly for Search also works for Gemini training, provided the site allows its content to be used for training.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
When a Gemini user asks about a specific web page (for example, whether it says anything about the ruby HTML tag), the page is read at that moment rather than during crawling and may be used to ground the answer, which takes extra time.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Although Google's pipeline diagram shows rendering as part of indexing, Google's rendering is a detached system, kept separate because rendering is time-consuming and computationally expensive.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Google deduplicates pages because many sites have very many pages and Google's index does not have room for everything.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.
The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.
Google runs deduplication before feature extraction, so that the expensive extraction work is spent only on a trimmed-down, manageable set of documents.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
The structured data Google processes is not fed very differently to AI Overviews and AI Mode: after cleaning and quality work, the same data goes to both the classic results page and the AI features.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Sites need no extra work to appear in AI Overviews and AI Mode: both work on top of the existing web results, so what works for web results also works for the AI features.
AI Overviews and AI Mode are built on top of Search results: they are a different experience of the same content Google already has.
Users often assume AI is all-knowing and borderless, but AI is still language-dependent: if an AI answer is synthesized from the top results, a query in another language draws on a totally different set of data.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Among the many signals Google calculates during indexing, the ones singled out as having large effects on search results were country, language, freshness, SafeSearch and spam.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
SpamBrain, Google's AI-based spam detection system, is nowadays built on Gemini and fine-tuned specifically for finding spam.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Index selection is a predictive AI system that relies heavily on machine learning.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
Index selection is the last step before documents enter Google's index.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Each document in Google's index has pretty much all the signals calculated for it attached, for example quality signals plus the page's country and language, according to an illustration the speaker called an approximation of the real structure.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
AI Overviews and AI Mode use the same index structures and token-based snippets as classic web results, a point Google called important but not obvious.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Google's slide on how LLM features with grounding generally work showed a query going to both the search engine and an LLM, the search engine's results going to the LLM, the LLM generating fan-out queries that go back to the search engine, and the LLM returning answers with links.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Which kinds of results Google shows for a query is decided by query understanding, which tries to predict the intent behind the query.
Google placed query understanding in the traditional part of Search, the part where answers are looked up in an index and then presented and ranked.
Google's closing slide said AI on Google is just SEO: AI features on Google Search use exactly the same processes as traditional results, so no new acronym is needed, as none was for mobile-first indexing or structured data.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
AI Overviews and AI Mode are built on top of Search results: they are a different experience of the same content Google already has.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Google's index is immense but finite, so Google cannot index every URL it finds on a web with a practically infinite number of URLs.
For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.
The speaker recapped Google's pipeline up to the index: Google crawls pages, processes the fetched documents and then stores them in its index.
Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.
Opening Day 2, Google recapped that Day 1 began with how Search works and then covered crawling, including basics of how the internet works such as TCP/IP, so that attendees could understand how crawlers work.
Google's crawling talk opened with how the internet works (TCP/IP, internet service providers, IP addresses, URLs, DNS and HTTP), because crawling can only be understood with that wider scope.
Google's serving diagram shows the query passing through query understanding and retrieval to the index, then back through ranking and search features to the user, with the Search Features step highlighted (example query 'Where to eat orange').
Google's opening slide for serving day showed serving as the third stage after crawling and indexing, and drew Google's serving infrastructure as query understanding and retrieval leading into the index, then ranking and search features leading back to the user, for the example query 'Where to eat jamon'.
AI Overviews and AI Mode are built on the Search infrastructure Google has used for 25 to 30 years and have very few processes of their own.
AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.
AI Overviews and AI Mode are built on the Search infrastructure Google has used for 25 to 30 years and have very few processes of their own.
AI Overviews and AI Mode use the same index structures and token-based snippets as classic web results, a point Google called important but not obvious.
Keep pages indexable and snippet-eligible, the whole technical requirement for AI features
Rests on 5 claims, 1 of them in this topic
Give every indexable page a unique, descriptive title element in the server HTML
Rests on 7 claims, 1 of them in this topic
Plan releases and fixes around Google's processing times, and judge results only after them
Rests on 11 claims, 1 of them in this topic
Also instory angle A-001