Processing: from fetched page to index

Day 1: Crawling 2

Said on stage 2

Day 2: Indexing 34

Shown on screen 16

Said on stage 11

What Google's documentation says 4

Analysis by the author 3

Day 3: Serving: Ranking, Search Console, and Performance 4

Shown on screen 2

Said on stage 2

Across days and sessions 35

1. Slide D2-C024 Day 2 · How is HTML interpreted

   A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.

   extends
   Slide D1-C063 Day 1 · How crawling works

   The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

2. Slide D2-C024 Day 2 · How is HTML interpreted

   A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.

   extends
   Slide D1-C064 Day 1 · How crawling works

   The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

3. Slide D2-C024 Day 2 · How is HTML interpreted

   A Google slide titled Data from Crawling showed an example of what the crawler passes on for processing: the fetch result, the connect time and the time to first byte in milliseconds, the robots policies that apply to the fetch, and the raw HTTP response with the page's HTML.

   extends
   Slide D1-C092 Day 1 · How Google thinks about crawl budget

   Hostload is driven by changes in connect time, changes in time to first byte, and HTTP 429 or 5xx status codes. If these increase, hostload is adjusted and crawling slows down.

4. Slide D2-C025 Day 2 · How is HTML interpreted

   In Google's example fetch record, the robots policies that apply to a fetch, shown as the value opt-out-google-extended, travel with the fetched page into processing.

   extends
   Slide D1-C064 Day 1 · How crawling works

   The crawler has multiple tasks: fetch from the internet, ensure it doesn't break the internet, and enforce robots.txt policies. Fetched data is sent for indexing.

5. Slide D2-C025 Day 2 · How is HTML interpreted

   In Google's example fetch record, the robots policies that apply to a fetch, shown as the value opt-out-google-extended, travel with the fetched page into processing.

   extends
   Docs D1-C086 Day 1 · How Google interprets robots.txt

   Google-Extended is a control token, not a crawler with its own user agent string. It decides whether crawled content may be used to train future Gemini models and to ground Gemini apps and Vertex AI.

6. Slide D2-C025 Day 2 · How is HTML interpreted

   In Google's example fetch record, the robots policies that apply to a fetch, shown as the value opt-out-google-extended, travel with the fetched page into processing.

   extends
   Stage D1-C522 Day 1 · How Google interprets robots.txt

   Google said a site can keep a directory or the whole site out of Gemini model training by disallowing it for the Google-Extended token in robots.txt, and that Google respects that policy; the speaker believed Google was the first to offer such an opt-out from model training.

7. Slide D2-C026 Day 2 · How is HTML interpreted

   A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

   extends
   Slide D1-C036 Day 1 · How Search works and where's AI?

   Search runs as three stages, crawling, indexing and serving, and the event covered one stage per day.

8. Slide D2-C026 Day 2 · How is HTML interpreted

   A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

   extends
   Slide D1-C063 Day 1 · How crawling works

   The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

9. Slide D2-C031 Day 2 · How is HTML interpreted

   Google extracts the rel=canonical link, through which site owners state their preferred canonical URL, and uses it in deduplication and in canonical selection.

   extends
   Docs D1-C128 Day 1 · session not recorded

   Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

10. Slide D2-C048 Day 2 · How is HTML interpreted

    Links extracted during processing are sent back to the crawl queue, where scheduling starts again for the newly found URLs.

    extends
    Slide D1-C063 Day 1 · How crawling works

    The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

11. Slide D2-C127 Day 2 · Lightning session D: Rendering and JavaScript

    A Google pipeline diagram ran from the crawl queue to the crawler, then through HTML parsing to processing, which passes pages to rendering and gets them back, and then to the index; rendering fetches its JavaScript and CSS through the crawler.

    extends
    Slide D1-C063 Day 1 · How crawling works

    The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

12. Stage D2-C128 Day 2 · Lightning session D: Rendering and JavaScript

    Although Google's pipeline diagram shows rendering as part of indexing, Google's rendering is a detached system, kept separate because rendering is time-consuming and computationally expensive.

    extends
    Slide D2-C026 Day 2 · How is HTML interpreted

    A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

13. Slide D2-C167 Day 2 · Lightning session D: Rendering and JavaScript

    The diagram from Google's JavaScript SEO basics page shows a URL going from the crawl queue to the crawler, the crawled HTML going to processing, then the render queue and the renderer, whose rendered HTML returns to processing before the page reaches the index.

    extends
    Slide D1-C063 Day 1 · How crawling works

    The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

14. Slide D2-C168 Day 2 · Lightning session D: Rendering and JavaScript

    In Google's processing step, links are extracted from the HTML Google already has, before rendering, and the URLs found go back to the crawl queue.

    extends
    Slide D1-C040 Day 1 · How Search works and where's AI?

    URL discovery works through links: a homepage links to section pages, which link to further pages.

15. Slide D2-C168 Day 2 · Lightning session D: Rendering and JavaScript

    In Google's processing step, links are extracted from the HTML Google already has, before rendering, and the URLs found go back to the crawl queue.

    extends
    Slide D2-C048 Day 2 · How is HTML interpreted

    Links extracted during processing are sent back to the crawl queue, where scheduling starts again for the newly found URLs.

16. Docs D2-C169 Day 2 · Lightning session D: Rendering and JavaScript

    Google's JavaScript SEO basics guide says Googlebot extracts links twice, from the HTML response before rendering and again from the rendered HTML, so links injected with JavaScript can be found if they use crawlable <a href> markup.

    extends
    Slide D1-C040 Day 1 · How Search works and where's AI?

    URL discovery works through links: a homepage links to section pages, which link to further pages.

17. Stage D2-C379 Day 2 · Handling web duplication

    rel=canonical is probably the most common way site owners signal duplicates, but it is very often wrong, for example a tag whose value reads 'canonical target' instead of a real URL (the example is partly unclear in the recording), so Google can only sometimes trust it.

    extends
    Slide D2-C031 Day 2 · How is HTML interpreted

    Google extracts the rel=canonical link, through which site owners state their preferred canonical URL, and uses it in deduplication and in canonical selection.

18. Slide D2-C382 Day 2 · Handling web duplication

    When localized pages are clustered, Google tries to use hreflang alternates; the speaker's short version of the advice was to use hreflang, which he called really helpful for same-language, different-country content.

    extends
    Slide D2-C033 Day 2 · How is HTML interpreted

    Google extracts hreflang annotations, through which site owners specify the language variants of their content, to know whether a page has an equivalent with similar content in another language.

19. Slide D2-C441 Day 2 · Finding the gold nuggets: structured data, media, and more!

    Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.

    extends
    Slide D1-C063 Day 1 · How crawling works

    The crawl pipeline runs from a crawl queue to a scheduler to the crawler, which fetches from the internet and passes the fetch reply to indexing.

20. Stage D2-C444 Day 2 · Finding the gold nuggets: structured data, media, and more!

    Google runs deduplication before feature extraction, so that the expensive extraction work is spent only on a trimmed-down, manageable set of documents.

    extends
    Slide D2-C026 Day 2 · How is HTML interpreted

    A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

21. Stage D2-C474 Day 2 · What is Structured Data and why we need it on the internet.

    Google's systems sometimes fail to determine a page's main content correctly, and structured data helps because site owners tend to mark up what is actually important rather than boilerplate or ads.

    extends
    Slide D2-C028 Day 2 · How is HTML interpreted

    Google extracts every element of a page so that it can tell the header, the navigation and the main content apart, and a slide labelled the main content as very important.

22. Stage D2-C688 Day 2 · Deciding what goes in the index?

    Index selection is the last step before documents enter Google's index.

    extends
    Stage D1-C211 Day 1 · How Search works and where's AI?

    Index selection runs after signals are collected and duplicates are dropped, and decides what goes into Google's index, which is big but not limitless.

23. Stage D2-C688 Day 2 · Deciding what goes in the index?

    Index selection is the last step before documents enter Google's index.

    extends
    Slide D2-C026 Day 2 · How is HTML interpreted

    A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

24. Stage D2-C861 Day 2 · Understanding what's on a page

    Gary Illyes said that whatever a site puts in its navigation or header tells Google the site does not particularly care about that content: it may help users do something on the side, but it is not what the page wants them to do, read or take away.

    extends
    Slide D2-C309 Day 2 · Understanding what's on a page

    A Google slide labelled the parts of an example blog page, marking the header and the navigation as not so important and the main content as very important.

25. Stage D2-C868 Day 2 · Understanding what's on a page

    Gary Illyes said the main content is what Google considers when ranking a page.

    extends
    Slide D2-C028 Day 2 · How is HTML interpreted

    Google extracts every element of a page so that it can tell the header, the navigation and the main content apart, and a slide labelled the main content as very important.

26. Stage D2-C930 Day 2 · Using images to your advantage and Engaging Search users with videos

    Google's media indexer processes the images and videos that feature extraction passes to it and attaches them to the URL of the page that hosts them.

    extends
    Stage D2-C448 Day 2 · Finding the gold nuggets: structured data, media, and more!

    For videos, Google's feature extraction takes the video itself and the data around it, to get a better sense of what happens in the video, and passes them to Google's media indexing engine.

27. Stage D3-C323 Day 3 · How Search results are born

    Most of Google's search features need nothing extra from the site owner; Google generates them from what it extracted from the page during indexing.

    extends
    Stage D2-C446 Day 2 · Finding the gold nuggets: structured data, media, and more!

    The 'gold nuggets' that Google's feature extraction step pulls out of a page's HTML are structured data (such as JSON-LD), images and videos.

28. Stage D3-C626 Day 3 · How long does it take to..?

    Google renders pages in two ways: immediately after crawling, or later through a queue-based process that runs elsewhere.

    extends
    Slide D2-C170 Day 2 · Lightning session D: Rendering and JavaScript

    After processing, an indexable page is placed in Google's render queue to wait for rendering.

29. Analysis D3-C676 Day 3 · How long does it take to..?

    The stage doubt that every URL gets rendered sits beside Google's JavaScript guide, which says every page with a 200 status is queued for rendering unless a robots rule blocks indexing: queued is not the same as rendered, so do not rely on rendering for critical content.

    extends
    Docs D2-C131 Day 2 · Lightning session D: Rendering and JavaScript

    Google's JavaScript SEO guide says Googlebot sends every page with a 200 HTTP status code to the rendering queue, whether or not it contains JavaScript, unless a robots meta tag or header tells Google not to index it, and Google uses the rendered HTML to index the page.

30. Slide D2-C048 Day 2 · How is HTML interpreted

    Links extracted during processing are sent back to the crawl queue, where scheduling starts again for the newly found URLs.

    repeats
    Stage D1-C328 Day 1 · How crawling works

    During indexing Google extracts the URLs found in fetched content and passes them back to the crawler and the scheduler to be crawled.

31. Slide D2-C262 Day 2 · What is Google friendly JavaScript

    Content that is not in the final DOM after rendering cannot be seen by Google, so it cannot be indexed.

    repeats
    Slide D2-C174 Day 2 · Lightning session D: Rendering and JavaScript

    Whatever is in the DOM at the moment Google's rendering finishes is what likely gets indexed.

32. Slide D2-C309 Day 2 · Understanding what's on a page

    A Google slide labelled the parts of an example blog page, marking the header and the navigation as not so important and the main content as very important.

    repeats
    Slide D2-C028 Day 2 · How is HTML interpreted

    Google extracts every element of a page so that it can tell the header, the navigation and the main content apart, and a slide labelled the main content as very important.

33. Slide D2-C441 Day 2 · Finding the gold nuggets: structured data, media, and more!

    Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.

    repeats
    Slide D2-C026 Day 2 · How is HTML interpreted

    A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

34. Slide D2-C442 Day 2 · Finding the gold nuggets: structured data, media, and more!

    Google's indexing pipeline slide drew a feedback line from the Processing stage back to the crawl queue.

    repeats
    Slide D2-C048 Day 2 · How is HTML interpreted

    Links extracted during processing are sent back to the crawl queue, where scheduling starts again for the newly found URLs.

35. Stage D2-C524 Day 2 · Using images to your advantage and Engaging Search users with videos

    Google extracts images and videos from the page's document object model (DOM), and finds images with a fairly standard HTML parser that looks for img elements.

    repeats
    Stage D2-C447 Day 2 · Finding the gold nuggets: structured data, media, and more!

    For images, Google's feature extraction takes the img element with its src and other attributes, including inline images, and passes them on to Google's image indexing service.

Built on these claims 12

Developer requirements 12

• MustDocumentedDEV-CAN-03

  Put exactly one absolute rel=canonical in the server-rendered head of every indexable page

  Rests on 8 claims, 3 of them in this topic

• ShouldDocumentedDEV-HTM-01

  Wrap each page's primary content in one clearly delimited main area

  Rests on 14 claims, 2 of them in this topic

• ShouldDocumentedDEV-HTM-05

  Write valid HTML and close every element

  Rests on 4 claims, 2 of them in this topic

• ShouldDocumentedDEV-SDA-10

  Put the same structured data on duplicate URLs as on the canonical

  Rests on 3 claims, 2 of them in this topic

• MustDocumentedDEV-URL-01

  Make every navigational link an <a> element whose href holds a real URL

  Rests on 6 claims, 2 of them in this topic

• MustDocumentedDEV-HTM-03

  Give every indexable page a unique, descriptive title element in the server HTML

  Rests on 7 claims, 1 of them in this topic

• ShouldDocumentedDEV-IDX-12

  On sites with explicit content, keep it on its own subdomain or domain and label it with rating markup

  Rests on 4 claims, 1 of them in this topic

• MustDocumentedDEV-IMG-01

  Put every image that should be found in an <img> element with a src attribute

  Rests on 6 claims, 1 of them in this topic

• MustDocumentedDEV-MON-02

  Render every template in a headless browser before release and check it in URL Inspection right after

  Rests on 7 claims, 1 of them in this topic

• ShouldDocumentedDEV-PRF-01

  Keep connect time and time to first byte low and stable under crawler load

  Rests on 13 claims, 1 of them in this topic

• ShouldDocumentedDEV-REN-01

  Server-render the main content and everything indexing depends on (SSR, static generation or hybrid)

  Rests on 11 claims, 1 of them in this topic

• MustDocumentedDEV-VID-01

  Embed each video in a video, iframe, embed or object element that loads without user action

  Rests on 5 claims, 1 of them in this topic

Sources 15

• In-depth guide to how Google Search works Google Search Central · checked 3 October 2026
• List of Google's common crawlers Google · checked 4 October 2026
• What is canonicalization Google Search Central · checked 3 October 2026
• meta tags and attributes that Google supports Google Search Central · checked 3 October 2026
• How to specify a canonical URL with rel="canonical" and other methods Google Search Central · checked 3 October 2026
• Tell Google about localized versions of your page Google Search Central · checked 3 October 2026
• Understand the JavaScript SEO basics Google Search Central · checked 3 October 2026
• Block Search indexing with noindex Google Search Central · checked 3 October 2026
• Robots meta tag, data-nosnippet, and X-Robots-Tag specifications Google Search Central · checked 3 October 2026
• Use valid HTML to specify page metadata Google Search Central · checked 2 October 2026
• Crawling December: The how and why of Googlebot crawling Search Central blog (3 December 2024) · checked 2 October 2026
• Inside Googlebot: demystifying crawling, fetching, and the bytes we process Search Central blog (31 March 2026) · checked 3 October 2026
• Introduction to structured data markup in Google Search Google Search Central · checked 3 October 2026
• Google image SEO best practices Google Search Central · checked 3 October 2026
• Video SEO best practices Google Search Central · checked 3 October 2026