How does the index look like?

Links to other sessions 26

1. Stage D2-C722 Day 2 · How does the index look like?

   Each document in Google's index has pretty much all the signals calculated for it attached, for example quality signals plus the page's country and language, according to an illustration the speaker called an approximation of the real structure.

   extends
   Slide D1-C037 Day 1 · How Search works and where's AI?

   For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.

2. Stage D2-C726 Day 2 · How does the index look like?

   AI Overviews and AI Mode use the same index structures and token-based snippets as classic web results, a point Google called important but not obvious.

   extends
   Slide D1-C038 Day 1 · How Search works and where's AI?

   AI Mode and AI Overviews use the same crawling and the same index as Search. At serving they add grounding on the Search index and query fan-out.

3. Stage D2-C727 Day 2 · How does the index look like?

   Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).

   extends
   Slide D1-C051 Day 1 · How Search works and where's AI?

   Three reasons were given: generative AI features are built directly on the core ranking systems, query fan-out expands the original query to find related information, and generative AI features highlight content indexed by Google Search.

4. Stage D2-C727 Day 2 · How does the index look like?

   Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).

   extends
   Docs D1-C053 Day 1 · How Search works and where's AI?

   Query fan-out means running several related searches at once to gather more results; a question about lawn weeds may also search herbicides and weed prevention.

5. Stage D2-C727 Day 2 · How does the index look like?

   Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).

   extends
   Stage D1-C172 Day 1 · Welcome and opening keynotes

   Google said the Gemini model lets Search understand the user's intent, and query fan-out then adds further queries to the first one to enrich the quality of the answer.

6. Stage D2-C727 Day 2 · How does the index look like?

   Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).

   extends
   Stage D2-C073 Day 2 · Controlling indexing

   John Mueller said Google uses the snippet as a way of building AI Overviews and AI Mode answers, so if a page forbids a snippet, Google cannot use that snippet for them.

7. Docs D2-C729 Day 2 · How does the index look like?

   Google says query fan-out in AI Overviews and AI Mode issues related searches across subtopics and several data sources, which for AI Mode include the Knowledge Graph and shopping data as well as web content (AI Mode launch post, March 2025).

   extends
   Docs D1-C053 Day 1 · How Search works and where's AI?

   Query fan-out means running several related searches at once to gather more results; a question about lawn weeds may also search herbicides and weed prevention.

8. Stage D2-C741 Day 2 · How does the index look like?

   In embedding-based retrieval, the distance between the embeddings of documents and the embedding of the user's query decides which documents are returned.

   extends
   Docs D1-C129 Day 1 · How Search works and where's AI?

   Google's guide says creating separate content for every variation of how people might search, including fan-out queries, primarily to manipulate rankings or AI responses violates its scaled content abuse policy. It adds that its AI systems can understand a page's relevance even without an exact match to the query.

9. Stage D3-C013 Day 3 · Making sense of users' queries

   Google's query processing deliberately mirrors indexing: a query is transformed into something that can be matched against the index, and stop word removal is part of that transformation.

   extends
   Stage D2-C737 Day 2 · How does the index look like?

   A search query is broken into words with the same segmenter or tokenizer that Google used to build the index.

10. Stage D3-C059 Day 3 · Making sense of users' queries

    Google treats fan-out queries generated by the LLM the same way as queries typed by users, so understanding how normal queries work explains fan-out queries too.

    extends
    Stage D2-C727 Day 2 · How does the index look like?

    Google said that when AI Overviews or AI Mode run a query fan-out, the generated queries are sent to Google's Search index and documents come back with their snippets, which then feed the AI-generated answer (part of this passage is unclear in the recording).

11. Stage D3-C061 Day 3 · Making sense of users' queries

    Google tries to make fan-out queries distinct from each other for better coverage, avoiding asking the same question several times, which would return the same answers.

    extends
    Docs D2-C729 Day 2 · How does the index look like?

    Google says query fan-out in AI Overviews and AI Mode issues related searches across subtopics and several data sources, which for AI Mode include the Knowledge Graph and shopping data as well as web content (AI Mode launch post, March 2025).

12. Stage D3-C075 Day 3 · Making sense of users' queries

    At retrieval, Google splits the query into words, applies query understanding and expansion, and matches the words and their expansions against the posting lists.

    extends
    Stage D2-C736 Day 2 · How does the index look like?

    In posting-list retrieval, the posting lists of the query's words are intersected, which yields an unranked list of candidate URLs.

13. Stage D3-C075 Day 3 · Making sense of users' queries

    At retrieval, Google splits the query into words, applies query understanding and expansion, and matches the words and their expansions against the posting lists.

    extends
    Stage D2-C738 Day 2 · How does the index look like?

    At retrieval, Google looks up the posting lists of the query words that are actually important rather than of every word in the query.

14. Stage D3-C077 Day 3 · Making sense of users' queries

    The first condition for retrieving a document is that the query's words, or its concepts in the case of vectors or embeddings, are in the document or related to it.

    extends
    Stage D2-C740 Day 2 · How does the index look like?

    Besides posting lists, Google can retrieve documents through vector embeddings: parts of documents are associated with embeddings, which form a vector space used for retrieval.

15. Stage D3-C079 Day 3 · Making sense of users' queries

    To order candidates at retrieval, Google uses signals collected during indexing, and the first two are language and country.

    extends
    Stage D2-C722 Day 2 · How does the index look like?

    Each document in Google's index has pretty much all the signals calculated for it attached, for example quality signals plus the page's country and language, according to an illustration the speaker called an approximation of the real structure.

16. Stage D3-C083 Day 3 · Making sense of users' queries

    Google called quality the most important of the signals used to order candidates at retrieval: a URL of high quality is more likely to be retrieved from the index for specific queries.

    extends
    Stage D2-C722 Day 2 · How does the index look like?

    Each document in Google's index has pretty much all the signals calculated for it attached, for example quality signals plus the page's country and language, according to an illustration the speaker called an approximation of the real structure.

17. Stage D3-C311 Day 3 · How Search results are born

    Google handles an image used as a search query much like a text query interpreted as an embedding: the image is broken down into vectors (embeddings) that are then searched for in the index.

    extends
    Stage D2-C740 Day 2 · How does the index look like?

    Besides posting lists, Google can retrieve documents through vector embeddings: parts of documents are associated with embeddings, which form a vector space used for retrieval.

18. Stage D3-C316 Day 3 · How Search results are born

    Google generates the parts of a text result, such as title link and snippet, from its understanding of the underlying web page, even when the site owner provides nothing extra.

    extends
    Stage D2-C724 Day 2 · How does the index look like?

    The snippet shown for a web result is reconstructed from the tokens stored in Google's index: Google knows the position of each token in the document and rebuilds the snippet from those positions.

19. Stage D3-C325 Day 3 · How Search results are born

    AI Mode and AI Overviews are not rich results but standard search features: they need no structured data to function and work with the normal text results from Google's index.

    extends
    Stage D2-C726 Day 2 · How does the index look like?

    AI Overviews and AI Mode use the same index structures and token-based snippets as classic web results, a point Google called important but not obvious.

20. Stage D2-C719 Day 2 · How does the index look like?

    The speaker recapped Google's pipeline up to the index: Google crawls pages, processes the fetched documents and then stores them in its index.

    repeats
    Slide D1-C036 Day 1 · How Search works and where's AI?

    Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.

21. Stage D2-C720 Day 2 · How does the index look like?

    Google's Search index stores the tokens produced by tokenizing each page, as they are, together with the metadata attached to the tokens during tokenization.

    repeats
    Stage D2-C323 Day 2 · Understanding what's on a page

    When tokenizing for Search, Google attaches metadata to each token for use in ranking, such as whether the word appeared in the header or in the main content (tagged 'centerpiece' on the slide), in bold or in a heading; Gary Illyes said he was not showing all of the metadata.

22. Stage D2-C721 Day 2 · How does the index look like?

    Google's Search index does not hold the full content of pages; Google said storing full pages and pulling them out at serving time would be a very inefficient way of doing search.

    repeats
    Stage D2-C318 Day 2 · Understanding what's on a page

    Google does not store the complete sentences or the full HTML of a page in the Search index, because large pieces of text would be unsearchable; it tokenizes the text into the smallest segments that still allow search.

23. Stage D2-C737 Day 2 · How does the index look like?

    A search query is broken into words with the same segmenter or tokenizer that Google used to build the index.

    repeats
    Stage D2-C321 Day 2 · Understanding what's on a page

    For languages written without spaces, such as Thai and Chinese, Google uses exactly the same word segmentation when indexing a page as when interpreting the user's query, because otherwise the query could not be matched against the index.

24. Stage D3-C074 Day 3 · Making sense of users' queries

    Google's index uses posting lists: for each word, a list of the URLs associated with that word.

    repeats
    Stage D2-C733 Day 2 · How does the index look like?

    For most of the tokens Google finds on the web, though not every single one, the index keeps a posting list of the URLs that contain that token.

25. Stage D3-C076 Day 3 · Making sense of users' queries

    For retrieval, Google uses signals attached individually to each document in the index.

    repeats
    Stage D2-C722 Day 2 · How does the index look like?

    Each document in Google's index has pretty much all the signals calculated for it attached, for example quality signals plus the page's country and language, according to an illustration the speaker called an approximation of the real structure.

26. Stage D3-C702 Day 3 · Wrapping all up: AI, Search, and making sense of everything.

    AI Overviews and AI Mode are built on the Search infrastructure Google has used for 25 to 30 years and have very few processes of their own.

    repeats
    Stage D2-C726 Day 2 · How does the index look like?

    AI Overviews and AI Mode use the same index structures and token-based snippets as classic web results, a point Google called important but not obvious.