Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Indexing
Google said, in detail not in its documentation, that the Search index holds neither full pages nor sentences but tokens, the smallest searchable units (words, for languages written with spaces), each stored with its position and metadata, with posting lists of the URLs that contain most tokens. The metadata records where a word appeared (header, main content or 'centerpiece', bold, a heading; a further item, heard as title in one recording and italics in another, is left open) and spam signals such as white-on-white text, and snippets are rebuilt from the stored token positions. Thai, Chinese and other languages written without spaces are segmented with statistical models built from web content in that language, and queries go through the same segmenter so they match the index; retrieval then looks up only the query's important words. Day 2 partly contradicts Day 1's slide that Gemini shares tokenization with Search: Gary Illyes said AI-model tokenization is different ('or mostly'), and his slides showed Search keeping 'robots.txt' and 'tl;dr' whole where the AI tokenizer split them into sub-word pieces, as Google's Gemini documentation describes for long words (the slide also showed each token's numeric ID). Author’s view: read it as a shared step with different outputs, and treat leftover hidden keyword blocks as a liability, since spam metadata is stored with the tokens. Day 3 confirmed the mirror on the query side: Google transforms a query into something that can be matched against the index, removing stop words as part of that, while a phrase whose stop words matter is recognised as a whole and its words are indexed together (said at the event, not in Google's docs). Author’s view: a community speaker's picture of Googlebot seeing a page as ones and zeros blends two steps, since crawling fetches the page and tokenization happens later, when it is processed.
Things in this topic 6
Counts are claims that name the thing. All things
What to do
In tokenization for AI models, common English words stay whole and each maps to a numeric token ID, so the model works with IDs rather than with the words; on Google's slide the word 'can' had the same ID, 740, both times it appeared.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byglossary term Tokenization
Google's two tokenization slides showed the difference on the same sentence: the Search tokenizer kept 'robots.txt' and 'tl;dr' as single tokens, while the AI-model tokenizer split them into pieces such as 'tl' and 'dr' or 'robots' and 'txt', with the punctuation as separate tokens.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence 2 slide photos
When Google processes a page for indexing, it gives words different weights depending on the part of the page where they appear.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Used byrequirement DEV-HTM-02
Google does not store the complete sentences or the full HTML of a page in the Search index, because large pieces of text would be unsearchable; it tokenizes the text into the smallest segments that still allow search.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byglossary term Tokenization
For languages written with spaces between words, such as English and German, Search tokenization splits a sentence into its individual words.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Text in languages written without spaces, such as Thai and Chinese, would end up in the index as long strings that might never be searched for, so Google segments it into words with statistical models built from other web content in that language.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
For languages written without spaces, such as Thai and Chinese, Google uses exactly the same word segmentation when indexing a page as when interpreting the user's query, because otherwise the query could not be matched against the index.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Gary Illyes said a colleague, John, would cover how Google interprets the words of a query on the morning of Day 3.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
When tokenizing for Search, Google attaches metadata to each token for use in ranking, such as whether the word appeared in the header or in the main content (tagged 'centerpiece' on the slide), in bold or in a heading; Gary Illyes said he was not showing all of the metadata.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byrequirement DEV-HTM-07
Google also stores spam metadata with the tokens of a page, for example that text was white on a white background, so ranking can use that information.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence transcript
Used byrequirement DEV-HTM-06
Tokenization for AI models such as Gemini, in training and in inference, differs from tokenization for Search, although Gary Illyes qualified this with 'or mostly'.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Tokenizers for AI models split long words into sub-word pieces that may make no sense on their own, because a token for every possible word would make the vocabulary too big, and a generative model only cares about closeness in vector space.
Speaker Gary IllyesIn Day 2, 11:30 · Understanding what's on a pageEvidence slide photo, transcript
Used byglossary term Tokenization
Google's Search index stores the tokens produced by tokenizing each page, as they are, together with the metadata attached to the tokens during tokenization.
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
Used byrequirement DEV-HTM-07
Google's Search index does not hold the full content of pages; Google said storing full pages and pulling them out at serving time would be a very inefficient way of doing search.
“we don't have the full content of the page in our index”
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
The snippet shown for a web result is reconstructed from the tokens stored in Google's index: Google knows the position of each token in the document and rebuilds the snippet from those positions.
“the snippet that you see was reconstructed from these tokens”
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
For most of the tokens Google finds on the web, though not every single one, the index keeps a posting list of the URLs that contain that token.
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
Used byglossary term Posting list
A search query is broken into words with the same segmenter or tokenizer that Google used to build the index.
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
At retrieval, Google looks up the posting lists of the query words that are actually important rather than of every word in the query.
Speaker GoogleIn Day 2, 16:15 · How does the index look like?Evidence transcript
Day 1's slide said Gemini shares technologies such as tokenization with Search, while on Day 2 Gary Illyes showed that the two tokenizers split the same text differently ('or mostly'); read this as a shared processing step with different outputs, so Search's word tokens and Gemini's sub-word tokens are not the same units.
Author Ibrahim AnjroAnnotates Day 2, 11:30 · Understanding what's on a page
Hidden text is recorded at token level: Google stores spam metadata such as white-on-white text with the tokens, so leftover hidden keyword blocks are a liability, not neutral clutter, and should be removed.
Author Ibrahim AnjroAnnotates Day 2, 11:30 · Understanding what's on a page
Used byrequirement DEV-HTM-06
Google's query processing deliberately mirrors indexing: a query is transformed into something that can be matched against the index, and stop word removal is part of that transformation.
Speaker John MuellerIn Day 3, 10:25 · Making sense of users' queriesEvidence transcript
A community speaker compared Googlebot to an inspector who cannot enter the shop and only looks through the window, seeing the page as ones and zeros, which the speaker linked to tokenization.
Speaker Community speakersIn Day 3, 10:45 · Lightning session K: Facets of qualityEvidence transcript
The inspector metaphor blends two steps: Googlebot fetches pages during crawling, while tokenization happens later when the page is processed for the index, as Google explained on Day 2.
Author Ibrahim AnjroAnnotates Day 3, 10:45 · Lightning session K: Facets of quality
Tokenization for AI models such as Gemini, in training and in inference, differs from tokenization for Search, although Gary Illyes qualified this with 'or mostly'.
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
Google named Thai as a language that makes query understanding more complex because it does not separate words with spaces; the speaker added, hedging with 'apparently', that Thai uses spaces to separate sentences.
Text in languages written without spaces, such as Thai and Chinese, would end up in the index as long strings that might never be searched for, so Google segments it into words with statistical models built from other web content in that language.
Google's query processing deliberately mirrors indexing: a query is transformed into something that can be matched against the index, and stop word removal is part of that transformation.
For languages written without spaces, such as Thai and Chinese, Google uses exactly the same word segmentation when indexing a page as when interpreting the user's query, because otherwise the query could not be matched against the index.
Google's query processing deliberately mirrors indexing: a query is transformed into something that can be matched against the index, and stop word removal is part of that transformation.
A search query is broken into words with the same segmenter or tokenizer that Google used to build the index.
At retrieval, Google splits the query into words, applies query understanding and expansion, and matches the words and their expansions against the posting lists.
At retrieval, Google looks up the posting lists of the query words that are actually important rather than of every word in the query.
The inspector metaphor blends two steps: Googlebot fetches pages during crawling, while tokenization happens later when the page is processed for the index, as Google explained on Day 2.
Google does not store the complete sentences or the full HTML of a page in the Search index, because large pieces of text would be unsearchable; it tokenizes the text into the smallest segments that still allow search.
Google generates the parts of a text result, such as title link and snippet, from its understanding of the underlying web page, even when the site owner provides nothing extra.
The snippet shown for a web result is reconstructed from the tokens stored in Google's index: Google knows the position of each token in the document and rebuilds the snippet from those positions.
Google's Search index stores the tokens produced by tokenizing each page, as they are, together with the metadata attached to the tokens during tokenization.
When tokenizing for Search, Google attaches metadata to each token for use in ranking, such as whether the word appeared in the header or in the main content (tagged 'centerpiece' on the slide), in bold or in a heading; Gary Illyes said he was not showing all of the metadata.
Google's Search index does not hold the full content of pages; Google said storing full pages and pulling them out at serving time would be a very inefficient way of doing search.
Google does not store the complete sentences or the full HTML of a page in the Search index, because large pieces of text would be unsearchable; it tokenizes the text into the smallest segments that still allow search.
A search query is broken into words with the same segmenter or tokenizer that Google used to build the index.
For languages written without spaces, such as Thai and Chinese, Google uses exactly the same word segmentation when indexing a page as when interpreting the user's query, because otherwise the query could not be matched against the index.
Google's summary slide on query understanding noted that some languages do not use spaces between words, which complicates query understanding.
Text in languages written without spaces, such as Thai and Chinese, would end up in the index as long strings that might never be searched for, so Google segments it into words with statistical models built from other web content in that language.
Google's index uses posting lists: for each word, a list of the URLs associated with that word.
For most of the tokens Google finds on the web, though not every single one, the index keeps a posting list of the URLs that contain that token.
Never hide text from users, such as white-on-white or off-screen keyword blocks
Rests on 3 claims, 2 of them in this topic
Use real heading, title and emphasis elements instead of styling alone
Rests on 2 claims, 2 of them in this topic
Place the words a page should rank for in its main content, not in footers or sidebars
Rests on 5 claims, 1 of them in this topic