Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Topic · Crawling
On Day 1 community speaker Tobias Schwarz treated similar URLs, technically different URLs that differ in only a few characters, as an indicator of duplicate content, broken links, unnecessary redirects and other technical issues. He showed five kinds on well-known brand sites: a different protocol or host (HTTP vs HTTPS, www vs non-www), capitalisation, the number of delimiters such as slashes, a space encoded as %20 in one URL and + in another, and parameters in a different order. To find them he normalises a URL list twice, first without changing the URLs' meaning and then into a deliberately uniform form that groups similar URLs together. Because anyone can link to a variant, he advised that the application compute the expected URL for every request and redirect, or return an error, when the requested URL differs, including checks on unused or reordered parameters and on numeric IDs written with leading zeros. In the Day 1 Q&A Google likewise suggested working out the normalised URL and redirecting parameter variants to it, which costs crawl budget at first but leaves a clean slate, and a panelist said Tobias Schwarz's slides had pleased the engineer side of his brain. Author’s view: of the two answers, Google's canonicalization guide favours the redirect, a strong signal that its target should be canonical, while an error page throws away the links pointing at the variant.
Things in this topic 3
Counts are claims that name the thing. All things
What to do
A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11glossary term Similar URLs
In a community speaker's definition, similar URLs are technically different URLs that differ in only a small portion of their characters.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byglossary term Similar URLs
A community speaker showed five kinds of similar URLs on well-known brand sites: a different protocol or host (HTTP vs HTTPS, www vs non-www), different capitalisation, a different number of delimiters such as slashes, a space encoded as %20 in one URL and + in another, and parameters in a different order.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11glossary term Similar URLs
To find similar URLs in a URL list, a community speaker first applies basic normalisation that keeps the meaning: lowercase the scheme and host, remove default ports such as 80 for HTTP, resolve relative path segments and usually drop the fragment, which matters only to clients.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11
A community speaker then derives a uniform URL that deliberately changes the meaning (one scheme, everything lowercase, www and index files such as index.html removed, duplicate and trailing delimiters removed, parameters sorted alphabetically) and sorts the URL list by it, so that similar URLs group together.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Similar URLs usually come from programming errors or manually set links, but anyone can link to them from other sites, including malicious actors, so a site should be hardened against them.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11
A community speaker advised that a web application compute the expected URL for every request, for example with reverse routing from the page type and ID, and redirect or return an error page when the requested URL differs.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11
For query parameters, a community speaker advised checking that every parameter in a request is actually used and in the expected order, and otherwise redirecting to the expected URL with only the used parameters in the correct order.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Used byrequirement DEV-URL-11
A community speaker said most web applications wrongly accept a numeric ID in a URL written with leading zeros or as a decimal with trailing zeros, which creates further similar URLs, so such variants are worth testing.
Speaker Tobias SchwarzIn Day 1, 16:20 · Lightning session C: CrawlingEvidence transcript
Of the two answers a community speaker offered for a request to a URL variant (redirect, or an error page), Google's canonicalization guide favours the redirect: a redirect is a strong signal that its target should become canonical, while an error page throws away any links pointing at the variant.
Author Ibrahim AnjroAnnotates Day 1, 16:20 · Lightning session C: Crawling
For query parameters, a community speaker advised checking that every parameter in a request is actually used and in the expected order, and otherwise redirecting to the expected URL with only the used parameters in the correct order.
If faceted URLs must be crawled, use the standard & separator, keep filters in a consistent order, and return a 404 when a filter combination has no results.
Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.
A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.
One fix for parameter variants of a URL is to work out the normalised URL and redirect the variants to it: the redirects cost crawl budget at first, but leave the site with a clean slate.
A community speaker advised that a web application compute the expected URL for every request, for example with reverse routing from the page type and ID, and redirect or return an error page when the requested URL differs.
Compute the expected URL for every request and redirect variants to it
Rests on 8 claims, 6 of them in this topic