Search Central LiveDeep Dive Europe 2026

Knowledge base v2.13.0 · Community edition · data through 2 October 2026

Topic · Crawling

Crawl demand

Crawl demand is how much Google wants a site's URLs, and each Google crawler has its own. It rises with site quality, how often URLs change and how popular they are on the web. On Day 2 Google said site owners can influence 'Discovered – currently not indexed' by getting other URLs of the site indexed and showing that its content is useful to users. Google's robots meta tag specification adds that Googlebot considerably lowers the crawl rate of a URL after the date in its unavailable_after rule. Day 3 put demand in time: Google estimated that recrawling a known URL takes about 30 days on average, from seconds (for news sites, a likely reading of the recording) to never, since URLs not recrawled for a very long time are dropped from the index, and that a crawl demand update driven by Search takes about 20 hours (estimates not in Google's docs). When the web gets excited about a few URLs, Google raises crawl demand for the whole site, and other Google products such as Shopping create their own demand; increases in crawl capacity take one to three weeks, longer than decreases. An audience member's large client site showed recrawl frequency following site hierarchy, demand and internal linking, with pages clustered as soft 404s recrawled only every 160 to 190 days. The second recording of Day 1 added how the scheduler sets demand: it logs how often each page changes and crawls frequently changing pages first (a news homepage before its terms of service page), very likely deprioritises URLs or sites known to be historically spammy, and visits hub pages such as the homepage and category pages more often because they link to new or updated pages. The crawl budget talk said the quality that drives demand is that of the site as a whole, and that frequent crawling is a consequence of interest and quality, not a cause of better rankings. In the Q&A Google said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, it raises the site's crawl demand and crawls up to it as far as crawl capacity allows, which Day 3 repeated with Shopping as the example. It also said very large, constantly changing sites get no special handling: frequently changing URLs, or URLs useful to users, raise demand and Google tries to raise its capacity, the same logic as for small sites; news sites are crawled aggressively anyway; and, asked how a very large site can tell the two apart, that a drop in crawling is hard to attribute to lower demand or a lower capacity limit, although a capacity drop is most of the time an abrupt step down. Author’s view: read site quality as the baseline, with known URL-level signals still counting, as the same talk's slide and Google's crawl budget guide suggest.

Based on D1-C093, D1-C132, D2-C709, D2-C101, D2-C710, D3-C604, D3-C606, D3-C605, D3-C622, D3-C623, D3-C624, D3-C619, D3-C607, D3-C608, D1-C332, D1-C331, D1-C202, D1-C377, D1-C392, D1-C439, D1-C466, D1-C493, D1-C420, D1-C538, D1-C545

34 claims · raised in 6 sessions · said or shown on Day 1 and Day 2 and Day 3

Open in Reef mapOpen in Graph

Things in this topic 10

Counts are claims that name the thing. All things

What to do

  • Raise demand by improving or removing weak pages that are already indexed, rather than resubmitting URLs that wait to be crawled.
  • Use unavailable_after on pages with a known end date so Google spends less crawling on them once they expire.
  • Link important deep pages from strong hub pages and check recrawl intervals by site depth in server logs or the Crawl Stats report.
  • Read a sudden, abrupt step down in crawling as a likely capacity problem (server errors, slow responses, blocks) before looking at demand.

Day 1: Crawling 20

Shown on screen 1

SlideConsistent with docsD1-C093

Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript

Things

Used byglossary term Crawl demand

  • Extended by D1-C377 Day 1: The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual…
  • Extended by D1-C439 Day 1: Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to…
  • Extended by D2-C695 Day 2: Page quality is ultimately what decides whether a document is indexed, so focusing on quality is the most…
  • Extended by D2-C709 Day 2: Site owners can influence 'Discovered – currently not indexed' by getting other URLs of the site indexed and…
  • Extended by D3-C623 Day 3: When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the…

Said on stage 15

StageConsistent with docsD1-C202

Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they usually link out to new or updated pages.

Speaker Cherry PrommawinIn Day 1, 11:45 · How Search works and where's AI?Evidence transcript

Used byrequirement DEV-URL-04glossary term Hub pages

  • Extends D1-C040 Day 1: URL discovery works through links: a homepage links to section pages, which link to further pages.
  • Extended by D2-C820 Day 2: Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their…
StageConsistent with docsD1-C334

Different Google teams prioritise crawling differently: web search cares a lot about the quality of a site and its content, while Ads wants to check every publisher page that wants to appear in Google Ads, so it schedules those URLs as they come in.

Speaker Gary IllyesIn Day 1, 14:05 · How crawling worksEvidence transcript

  • Extended by D1-C538 Day 1: A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search…
StageNot in docsD1-C377

The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual page.

Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript

Things
  • Extends D1-C093 Day 1: Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on…
  • Extended by D3-C623 Day 3: When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the…
StageConsistent with docsD1-C392

Frequent crawling is usually a consequence, not a cause: Google visiting a site more often tends to signal that people are interested in the site and that its content is of high quality.

Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript

  • Answers D1-C390 Day 1: An audience member asked, in a question submitted before the event, whether crawl frequency affects the…
StageConsistent with docsD1-C424

Google's search crawling works to keep content fresh and to understand which pages change frequently, whereas many AI systems crawl a site with no understanding of it and take everything.

Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript

Used byrequirement DEV-AIF-05

  • Answers D1-C422 Day 1: An audience member asked, in a question submitted at registration, what the biggest difference is between…
StageD1-C438

An audience member asked how Google prioritises crawling and indexing for very large real-time sites, such as sports sites, whose content changes constantly and is largely near-duplicate across seasons.

From the audienceIn Day 1, 16:35 · Q&AEvidence transcript

  • Answered by D1-C439 Day 1: Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to…
  • Answered by D1-C440 Day 1: If a site's server cannot cope with the extra crawling that higher crawl demand brings, Google reduces its…
StageConsistent with docsD1-C439

Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.

Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript

Things
  • Answers D1-C438 Day 1: An audience member asked how Google prioritises crawling and indexing for very large real-time sites, such as…
  • Extends D1-C093 Day 1: Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on…
  • Extended by D1-C538 Day 1: A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search…
  • Extended by D3-C623 Day 3: When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the…
StageConsistent with docsD1-C447

Google crawls new URLs, such as plugin-generated calendar pages, because it wants to see what they are; its automated systems learn that such URLs are useless only from large samples, which takes time.

Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript

Used byrequirement DEV-URL-08

  • Extended by D1-C539 Day 1: A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example…
StageD1-C465

An audience member asked how a large news site can tell whether crawl budget is limiting how fast new articles are discovered (within minutes), and which statistics in Search Console and the logs show this.

From the audienceIn Day 1, 16:35 · Q&AEvidence transcript

Things
  • Answered by D1-C466 Day 1: Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively…
  • Answered by D1-C467 Day 1: To check that new content is crawled fast enough, Gary Illyes advised measuring the time between publishing a…
  • Answered by D1-C468 Day 1: When new URLs are picked up more slowly, Gary Illyes advised checking the log files for URLs that Google…
  • Answered by D1-C469 Day 1: Gary Illyes said disallowing a section that should not be crawled, such as /ads, in a Googlebot group in…
StageConsistent with docsD1-C466

Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.

Speaker Gary IllyesIn Day 1, 16:35 · Q&AEvidence transcript

Things

Used byrequirement DEV-MON-11

  • Answers D1-C465 Day 1: An audience member asked how a large news site can tell whether crawl budget is limiting how fast new…
  • Repeats D1-C381 Day 1: Not every site needs to worry about crawl budget, and this has always been so: a site with fewer than a few…
  • Extended by D3-C606 Day 3: For news sites in particular, Google said the refresh of a known URL can literally take seconds (the reading…
StageD1-C545

An audience member asked, for very large sites of about 100 million pages, what signs show that a site is limited by crawl budget, and how to tell a crawl capacity limit problem from a crawl demand problem.

From the audienceIn Day 1, 16:35 · Q&AEvidence transcript

Things
  • Answered by D1-C493 Day 1: There is no easy way to tell whether a drop in Google's crawling comes from lower crawl demand or a lower…
  • Answered by D1-C494 Day 1: To check whether Google lowered a site's crawl capacity limit, look at the number of connections Googlebot…
  • Answered by D1-C546 Day 1: A Google panelist illustrated a crawl capacity limit drop as a step down in the site's hostload, for example…
StageConsistent with docsD1-C493

There is no easy way to tell whether a drop in Google's crawling comes from lower crawl demand or a lower crawl capacity limit; most of the time a capacity-limit drop is an abrupt step down.

Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript

Things

Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)

  • Answers D1-C545 Day 1: An audience member asked, for very large sites of about 100 million pages, what signs show that a site is…
  • Extended by D3-C617 Day 3: Google's crawl chart puts a crawl capacity update at seconds when backing off, typically 4 hours or 1-2…
StageConsistent with docsD1-C538

A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.

Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript

Things

Used byglossary term Crawl demand

  • Extends D1-C439 Day 1: Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to…
  • Extends D1-C334 Day 1: Different Google teams prioritise crawling differently: web search cares a lot about the quality of a site…
  • Repeated by D3-C624 Day 3: Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate…

What Google's documentation says 1

DocsSourceD1-C132

Google's crawl budget guide says each crawler has its own crawl demand, but the crawl capacity limit (hostload) is shared across all crawlers, so high demand from one crawler can reduce the capacity left for others.

“high demand from one crawler can reduce the capacity available for others”

Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget

Things

Used byrequirement DEV-PRF-01glossary term Crawl rate limit (hostload)

Analysis by the author 3

AnalysisD1-C067

A page with no internal links depends on sitemaps alone to be found, so it is discovered slowly and attracts little crawl demand.

Author Ibrahim AnjroAnnotates Day 1, 14:05 · How crawling works

Things

Used byrequirements DEV-URL-04, DEV-URL-05

  • Extended by D2-C189 Day 2: Build market and language selectors as plain <a href> links to each alternate URL, not buttons or script…
  • Extended by D2-C427 Day 2: When internal links point only to page A and the canonical leader is reached only through A's canonical link…
AnalysisD1-C098

On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.

Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget

  • Extended by D2-C708 Day 2: The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on…
  • Extended by D2-C714 Day 2: 'Crawled – currently not indexed' is most of the time a quality issue rather than a technical one: the pages…
  • Extended by D2-C716 Day 2: Treat 'Crawled – currently not indexed' as a quality audit list: compare those URLs with indexed pages of the…
AnalysisD1-C420

The stage remark that crawl demand follows the quality of the site as a whole sits beside the same talk's slide (D1-C094), which falls back to the parent path's aggregate only when a URL's own quality is unknown, and Google's crawl budget guide lists page quality among the demand factors; read it as site quality setting the baseline while known URL-level signals still count.

Author Ibrahim AnjroAnnotates Day 1, 16:00 · How Google thinks about crawl budget

Things

Day 2: Indexing 4

Said on stage 1

StageConsistent with docsD2-C709

Site owners can influence 'Discovered – currently not indexed' by getting other URLs of the site indexed and showing Google's systems that the site's content is good and useful to users.

Speaker GoogleIn Day 2, 15:40 · Deciding what goes in the index?Evidence transcript

  • Extends D1-C093 Day 1: Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on…

What Google's documentation says 1

Analysis by the author 2

AnalysisD2-C708

The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.

Author Ibrahim AnjroAnnotates Day 2, 15:40 · Deciding what goes in the index?

Things

Used byrequirement DEV-MON-03

  • Extends D1-C098 Day 1: On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.

Day 3: Serving: Ranking, Search Console, and Performance 10

Shown on screen 2

Said on stage 7

StageConsistent with docsD3-C623

When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.

Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript

Things
  • Extends D1-C093 Day 1: Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on…
  • Extends D1-C377 Day 1: The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual…
  • Extends D1-C439 Day 1: Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to…
StageConsistent with docsD3-C624

Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate covers only demand from Search.

Speaker Gary IllyesIn Day 3, 15:45 · How long does it take to..?Evidence transcript

Things
  • Extends D1-C375 Day 1: Googlebot as a single standalone crawler is a historical idea: Google crawls through a centralised crawling…
  • Repeats D1-C538 Day 1: A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search…

Analysis by the author 1

Across days and sessions 23

  1. Stage D1-C202 Day 1 · How Search works and where's AI?

    Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they usually link out to new or updated pages.

    extends
    Slide D1-C040 Day 1 · How Search works and where's AI?

    URL discovery works through links: a homepage links to section pages, which link to further pages.

  2. Stage D1-C377 Day 1 · How Google thinks about crawl budget

    The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual page.

    extends
    Slide D1-C093 Day 1 · How Google thinks about crawl budget

    Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

  3. Stage D1-C439 Day 1 · Q&A

    Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.

    extends
    Slide D1-C093 Day 1 · How Google thinks about crawl budget

    Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

  4. Stage D1-C538 Day 1 · Q&A

    A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.

    extends
    Stage D1-C334 Day 1 · How crawling works

    Different Google teams prioritise crawling differently: web search cares a lot about the quality of a site and its content, while Ads wants to check every publisher page that wants to appear in Google Ads, so it schedules those URLs as they come in.

  5. Stage D1-C538 Day 1 · Q&A

    A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.

    extends
    Stage D1-C439 Day 1 · Q&A

    Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.

  6. Stage D1-C539 Day 1 · Q&A

    A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).

    extends
    Stage D1-C447 Day 1 · Q&A

    Google crawls new URLs, such as plugin-generated calendar pages, because it wants to see what they are; its automated systems learn that such URLs are useless only from large samples, which takes time.

  7. Analysis D2-C189 Day 2 · Lightning session D: Rendering and JavaScript

    Build market and language selectors as plain <a href> links to each alternate URL, not buttons or script handlers; otherwise the language versions have no internal links and depend on sitemaps to be found, which is slow.

    extends
    Analysis D1-C067 Day 1 · How crawling works

    A page with no internal links depends on sitemaps alone to be found, so it is discovered slowly and attracts little crawl demand.

  8. Stage D2-C427 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    When internal links point only to page A and the canonical leader is reached only through A's canonical link, the leader is reachable by machines but not by human visitors, a signal conflict that asks the search engine to index a page users cannot reach.

    extends
    Analysis D1-C067 Day 1 · How crawling works

    A page with no internal links depends on sitemaps alone to be found, so it is discovered slowly and attracts little crawl demand.

  9. Stage D2-C695 Day 2 · Deciding what goes in the index?

    Page quality is ultimately what decides whether a document is indexed, so focusing on quality is the most reliable way to get pages into Google's index.

    extends
    Slide D1-C093 Day 1 · How Google thinks about crawl budget

    Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

  10. Analysis D2-C708 Day 2 · Deciding what goes in the index?

    The help page explains 'Discovered – currently not indexed' by expected server overload (capacity), while on stage it was explained as Google not wanting the URL yet (demand); the crawl budget guide covers both, so first rule out slow responses and server errors, then treat the status as a quality and demand problem.

    extends
    Analysis D1-C098 Day 1 · How Google thinks about crawl budget

    On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.

  11. Stage D2-C709 Day 2 · Deciding what goes in the index?

    Site owners can influence 'Discovered – currently not indexed' by getting other URLs of the site indexed and showing Google's systems that the site's content is good and useful to users.

    extends
    Slide D1-C093 Day 1 · How Google thinks about crawl budget

    Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

  12. Stage D2-C714 Day 2 · Deciding what goes in the index?

    'Crawled – currently not indexed' is most of the time a quality issue rather than a technical one: the pages are usually low quality or useless for the index, for example duplicates or soft 404s.

    extends
    Analysis D1-C098 Day 1 · How Google thinks about crawl budget

    On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.

  13. Analysis D2-C716 Day 2 · Deciding what goes in the index?

    Treat 'Crawled – currently not indexed' as a quality audit list: compare those URLs with indexed pages of the same type for thin, duplicate or soft-404-like content and for template differences before looking for technical faults.

    extends
    Analysis D1-C098 Day 1 · How Google thinks about crawl budget

    On smaller sites, slow indexing is almost always a demand problem, meaning quality, not a capacity problem.

  14. Slide D2-C820 Day 2 · Welcome to indexing day!

    Google's Q&A slide advised large sites to rely on hub pages, such as category pages, that link out to their important pages, instead of on HTML sitemaps.

    extends
    Stage D1-C202 Day 1 · How Search works and where's AI?

    Google may visit hub pages, such as a homepage or category pages, more often than other pages, because they usually link out to new or updated pages.

  15. Stage D3-C606 Day 3 · How long does it take to..?

    For news sites in particular, Google said the refresh of a known URL can literally take seconds (the reading 'news sites' is likely but not certain).

    extends
    Stage D1-C466 Day 1 · Q&A

    Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.

  16. Slide D3-C612 Day 3 · How long does it take to..?

    Google may never fetch a lower-quality site's sitemap again: once it figures out the site is of lower quality, it no longer wants to fetch the sitemap.

    extends
    Stage D1-C331 Day 1 · How crawling works

    Google's crawl scheduler very likely deprioritises a URL when the URL or its site is known to be historically spammy.

  17. Slide D3-C617 Day 3 · How long does it take to..?

    Google's crawl chart puts a crawl capacity update at seconds when backing off, typically 4 hours or 1-2 weeks, and 1-3 weeks in recovery.

    extends
    Stage D1-C493 Day 1 · Q&A

    There is no easy way to tell whether a drop in Google's crawling comes from lower crawl demand or a lower crawl capacity limit; most of the time a capacity-limit drop is an abrupt step down.

  18. Stage D3-C623 Day 3 · How long does it take to..?

    When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.

    extends
    Slide D1-C093 Day 1 · How Google thinks about crawl budget

    Crawl demand is driven by the quality of the site, the change frequency of its URLs and their popularity on the internet.

  19. Stage D3-C623 Day 3 · How long does it take to..?

    When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.

    extends
    Stage D1-C377 Day 1 · How Google thinks about crawl budget

    The quality that drives crawl demand is the quality of the site as a whole, not the quality of an individual page.

  20. Stage D3-C623 Day 3 · How long does it take to..?

    When Google notices the web getting excited about a few URLs on a site, it allocates more crawl demand to the whole site so it does not miss content useful to future searchers.

    extends
    Stage D1-C439 Day 1 · Q&A

    Very large, constantly changing sites get no special handling: when URLs change frequently or are useful to users, Google raises crawl demand and tries to raise its crawl capacity for the site, the same logic it applies to small sites.

  21. Stage D3-C624 Day 3 · How long does it take to..?

    Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate covers only demand from Search.

    extends
    Stage D1-C375 Day 1 · How Google thinks about crawl budget

    Googlebot as a single standalone crawler is a historical idea: Google crawls through a centralised crawling infrastructure, so a request from a Google user agent in server logs is a request routed through that shared platform.

  22. Stage D1-C466 Day 1 · Q&A

    Gary Illyes said news sites rarely need to worry about crawl budget, because Google crawls them aggressively given their constant flow of new content and new URLs.

    repeats
    Stage D1-C381 Day 1 · How Google thinks about crawl budget

    Not every site needs to worry about crawl budget, and this has always been so: a site with fewer than a few thousand URLs likely has no crawl budget problem, and a larger site does not necessarily have one either.

  23. Stage D3-C624 Day 3 · How long does it take to..?

    Crawl demand can also come from other Google products, such as Shopping, and the roughly 20-hour estimate covers only demand from Search.

    repeats
    Stage D1-C538 Day 1 · Q&A

    A Google panelist said crawl demand is not only Search's: when enough of a site's URLs are wanted by Search, Google Ads or another Google product, Google raises the site's crawl demand and crawls up to that level of demand as far as the site's crawl capacity allows.

Built on these claims 9

Developer requirements 9

Sources 7