Search Central LiveDeep Dive Europe 2026

Knowledge base v2.13.0 · Community edition · data through 2 October 2026

Topic · Crawling

HTTP caching for crawlers

Supporting conditional requests lets Google skip re-downloading unchanged pages. Answer with 304 Not Modified, and prefer ETag. Day 1's second recording added that Gary Illyes named Last-Modified and ETag as the response headers that matter for caching, and that Cherry Prommawin described a 304 as telling Google's crawler that the content has not changed since its last visit.

Based on D1-C104, D1-C103, D1-C321, D1-C350

5 claims · raised in 3 sessions · said or shown on Day 1

Open in Reef mapOpen in Graph

Things in this topic 4

Counts are claims that name the thing. All things

What to do

  • Check that the server returns 304 to If-None-Match and If-Modified-Since requests.

Day 1: Crawling 4

Shown on screen 1

SlideConsistent with docsD1-C103

Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.

Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence 2 slide photos, transcript

Things

Used byrequirements DEV-SRV-08, DEV-URL-08

  • Extended by D1-C379 Day 1: The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages…
  • Extended by D2-C886 Day 2: A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster…

Said on stage 2

What Google's documentation says 1

DocsSourceD1-C104

HTTP caching for crawlers means supporting conditional requests (ETag with If-None-Match, or Last-Modified with If-Modified-Since) and answering 304 Not Modified when nothing changed. Google's crawling team has said it prefers ETag.

Publisher Google, Search Central blog (9 December 2024)Annotates Day 1, 16:00 · How Google thinks about crawl budget

Used byrequirement DEV-SRV-08

Day 2: Indexing 1

What Google's documentation says 1

DocsSourceD2-C829

Google's guide to fixing Search-related JavaScript problems says the Web Rendering Service may ignore caching headers and so use outdated JavaScript or CSS, and recommends content fingerprinting, which puts a hash of the content in the file name, as in main.2bb85551.js.

Publisher Google Search CentralAnnotates Day 2, 11:15 · What is Google friendly JavaScript

Things

Used byrequirement DEV-REN-06

Across days and sessions 2

  1. Stage D1-C379 Day 1 · How Google thinks about crawl budget

    The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.

    extends
    Slide D1-C103 Day 1 · How Google thinks about crawl budget

    Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.

  2. Stage D2-C886 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster loading and no crawl budget spent on content that no longer mattered.

    extends
    Slide D1-C103 Day 1 · How Google thinks about crawl budget

    Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.

Built on these claims 3

Developer requirements 3

Sources 5