Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
Filter and sort URLs multiply into near-infinite combinations and are a classic crawl budget leak. Google prefers blocking them in robots.txt and keeping only item pages and one unfiltered listing crawlable. Day 2 added that non-crawlable link markup, such as onclick handlers and hash pseudo-links, is common in single-page apps, so sites with faceted navigation should check their links for it. Author’s view: hash fragments can deliberately keep filter states out of the crawl, as Google's faceted navigation guide allows, as long as product, category and language links use real URLs. Day 1's second recording added that Google crawls every URL that differs even when it leads to the same content, so infinite URL spaces burn crawl budget; a community speaker advised checking that every parameter in a request is used and in the expected order and redirecting otherwise, and in the Q&A Google suggested working out the normalised URL and redirecting parameter variants to it, which costs crawl budget at first but leaves a clean slate, said log files are worth checking on sites with filter parameters, and noted that many crawling complaints trace back to plugins that generate URLs, such as calendars, whose useless parameter URLs can be handled at the web server.
Things in this topic 9
Counts are claims that name the thing. All things
What to do
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo, transcript
Used byrequirement DEV-URL-08
Six faceted-URL problems were shown: the same facets in a different order, irrelevant or conflicting facet combinations, excessive facet selection, facets on paginated series, facets combined with search queries, and optional facets with default values.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence slide photo
Used byrequirement DEV-URL-08glossary term Faceted navigation
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence 2 slide photos, transcript
Used byrequirements DEV-SRV-08, DEV-URL-08
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
Speaker Cherry PrommawinIn Day 1, 16:00 · How Google thinks about crawl budgetEvidence transcript
Used byrequirements DEV-URL-08, DEV-URL-11
For query parameters, a community speaker advised checking that every parameter in a request is actually used and in the expected order, and otherwise redirecting to the expected URL with only the used parameters in the correct order.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11
Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
One fix for parameter variants of a URL is to work out the normalised URL and redirect the variants to it: the redirects cost crawl budget at first, but leave the site with a clean slate.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-11
Many crawling complaints that reach Google's crawling team turn out to be caused by a plugin, for example a WordPress calendar plugin that adds calendar parameters to the URLs of every post, page, category and tag page.
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
Speaker not identifiedIn Day 1, 16:35 · Q&AEvidence transcript
Used byrequirement DEV-URL-08
Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt and keep crawlable only item pages plus one unfiltered listing page, or use URL fragments, which Google generally does not crawl. rel=canonical and nofollow are weaker, slower options.
Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirement DEV-URL-08glossary term Faceted navigation
If faceted URLs must be crawled, use the standard & separator, keep filters in a consistent order, and return a 404 when a filter combination has no results.
Publisher GoogleAnnotates Day 1, 16:00 · How Google thinks about crawl budget
Used byrequirement DEV-URL-09
Non-crawlable link markup, such as onclick links and hash pseudo-links, is common in single-page web apps, and sites that use faceted navigation should check their links for it.
Speaker Rebecca YuIn Day 2, 10:40 · Lightning session D: Rendering and JavaScriptEvidence transcript
Used byrequirement DEV-URL-02glossary term Single-page app (SPA)
Hash-fragment links are a problem only where Google should follow them: product, category and language links need a real URL in an <a href>, while fragments can deliberately keep filter combinations out of the crawl, as Google's faceted navigation guide allows.
Author Ibrahim AnjroAnnotates Day 2, 10:40 · Lightning session D: Rendering and JavaScript
Used byrequirement DEV-URL-08
Google treats URLs that differ as different URLs and crawls all of them even when they lead to the same content, so infinite URL spaces such as calendars or many versions of a page burn crawl budget.
Three things burn crawl budget: server errors unrelated to server load, useless pages and resources, and infinite URL spaces.
The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
For query parameters, a community speaker advised checking that every parameter in a request is actually used and in the expected order, and otherwise redirecting to the expected URL with only the used parameters in the correct order.
If faceted URLs must be crawled, use the standard & separator, keep filters in a consistent order, and return a 404 when a filter combination has no results.
Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.
A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
A Google panelist said useless plugin-generated parameter URLs can be handled at the web server, for example with a rule or a custom module in Apache, which saves crawl budget so that Google may pick up the URLs that matter instead (the exact mechanism is not clear in the recording).
Hash-fragment links are a problem only where Google should follow them: product, category and language links need a real URL in an <a href>, while fragments can deliberately keep filter combinations out of the crawl, as Google's faceted navigation guide allows.
Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt and keep crawlable only item pages plus one unfiltered listing page, or use URL fragments, which Google generally does not crawl. rel=canonical and nofollow are weaker, slower options.
A community speaker listed the technical SEO goals of a two-site consolidation as lighter pages, faster loading and no crawl budget spent on content that no longer mattered.
Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.
One fix for parameter variants of a URL is to work out the normalised URL and redirect the variants to it: the redirects cost crawl budget at first, but leave the site with a clean slate.
A community speaker advised that a web application compute the expected URL for every request, for example with reverse routing from the page type and ID, and redirect or return an error page when the requested URL differs.
URL fragments (#) are often ignored by crawlers: a fragment exists only in the browser, so Google cannot request it.
Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt and keep crawlable only item pages plus one unfiltered listing page, or use URL fragments, which Google generally does not crawl. rel=canonical and nofollow are weaker, slower options.
Keep faceted navigation out of the crawl with robots.txt rules or URL fragments
Rests on 12 claims, 9 of them in this topic
Compute the expected URL for every request and redirect variants to it
Rests on 8 claims, 3 of them in this topic
Support conditional requests with ETag or Last-Modified and answer 304 when nothing changed
Rests on 5 claims, 1 of them in this topic
Do not navigate with onclick handlers, javascript: URLs, routerLink without href or # pseudo-links
Rests on 6 claims, 1 of them in this topic
If facet URLs must be crawlable, use & separators, a fixed filter order and 404 for empty results
Rests on 5 claims, 1 of them in this topic