Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Thing · Concept
The same or nearly the same content at several URLs, which Google deduplicates: it clusters the URLs and serves one canonical; not a penalty.
Glossary · Duplicate cluster
A group of pages Google considers essentially equivalent: it stores one in the index and keeps track of the other URLs.
Documented in
Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.
Google Search Central · Day 1 · session not recorded
Google's canonicalization troubleshooting guide says fixing a wrong duplicate cluster comes down to making the clustered pages sufficiently different; pages split out faster when the difference is clear and significant, and Google may keep pages in a duplicate cluster for up to two weeks after a fix.
Google Search Central · Day 2 · Handling web duplication
Google's documentation says a search result usually points to the canonical page, but the other pages in a duplicate cluster are alternate versions that may be served in different contexts, for example a mobile page for a user on a mobile device.
Google Search Central · Day 2 · Deciding what goes in the index?
Google's guide to fixing canonicalization issues says that even after content issues are fixed, Google might hold pages in a duplicate cluster for up to two weeks, and that pages split out faster when they differ clearly and significantly.
Google Search Central · Day 3 · How long does it take to..?
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Storage is a second reason for deduplication: Google's storage has many competing uses and storage prices have risen sharply, so the space for any one use is limited and Google has to draw a line somewhere.
City pages can trigger the same pattern-based deduplication: for a car dealer brand with branches in several cities and similar stock, Google's systems may decide the city name does not matter and canonicalise to one city's page; the slide asked whether a further city page such as /zurich/services would be treated the same way.
Google's speaker said that pointing rel=canonical from the pages of a paginated set to the first page can sometimes make sense depending on the goal, for example to make a category page more visible, but that it affects canonicalization and deduplication.
Google's indexing pipeline slide put a Processing stage between the crawler and the index and listed HTML parsing, rendering, deduplication and feature extraction above it, with feature extraction highlighted as the step of this talk.
Gary Illyes · Day 2 · Finding the gold nuggets: structured data, media, and more!
Google runs deduplication before feature extraction, so that the expensive extraction work is spent only on a trimmed-down, manageable set of documents.
Gary Illyes · Day 2 · Finding the gold nuggets: structured data, media, and more!
Gemini is not part of Search, but it uses crawlers for data, shares some technologies such as tokenization and deduping, and grounds on the Search index.
When a page is selected for Google's index, its whole duplicate cluster goes into the index with it.
Cherry Prommawin · Day 1 · How Search works and where's AI?
A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.
Tobias Schwarz · Day 1 · Lightning session C: Crawling
Gary Illyes said there is no such thing as a duplicate content penalty.
Gary Illyes · Day 1 · session not recorded
A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.
Google deduplicates pages because many sites have very many pages and Google's index does not have room for everything.
For deduplication, a cluster is a set of pages Google considers essentially equivalent: Google stores one of them in the index and keeps track of the other related URLs.
Google treats a site migration as deduplication across sites, in which the site owner says the old and the new domain are the same and that Google should pick the new domain, so deduplication also helps Google handle migrations.
Google's duplication talk described three related parts of deduplication: building clusters, localization, and selecting the representative URL, which is the canonicalization site owners see in Search Console.
Google builds duplicate clusters from four kinds of input: redirects, content, rel=canonical, and a 'magic bucket' of other things.
To avoid pattern-based deduplication, Google's speaker recommended not having many unrelated, similar-looking URLs that lead to the same content, and returning error pages for URLs that no longer exist so they are clearly unrelated.
Same-language content for different countries is tricky for Google's deduplication, notably German pages for Germany, Austria and Switzerland, and possibly Spanish-language variants.
Google extracts the rel=canonical link, through which site owners state their preferred canonical URL, and uses it in deduplication and in canonical selection.
Google's deduplication has three steps: identify and cluster duplicate web pages, pick representative URLs and index the unique pages, and forward signals to the representative URLs.
Google forwards the signals attached to every URL in a duplicate cluster, such as links, to the representative URL, so that nothing is lost by showing only one URL.
The first reason Google deduplicates is that users do not want to see the same page repeated in the search results, even if site owners would like it to rank ten times on page one.
Google keeps the other URLs of a duplicate cluster as 'alternate names': equivalent URLs with the same content that Google still tracks as alternate versions of the representative URL.
Google's slide said there is not one single ranking system and named spam detection systems, the reviews system, BERT, MUM, RankBrain, freshness systems, deduplication systems, crisis information systems and link analysis systems (PageRank).
Because Google extracts structured data, images and videos only after deduplication, put markup and media on the URL you want as canonical and keep them identical on its duplicates; markup that exists only on a duplicate that loses canonical selection may never be extracted.
Ibrahim Anjro · Day 2 · Finding the gold nuggets: structured data, media, and more!
Kit items about Duplicate content: their own words name it, or several of the claims they rest on do.
Make pages that share a URL pattern differ in their main content
Rests on 6 claims, 3 of them naming Duplicate content
Put the same structured data on duplicate URLs as on the canonical
Rests on 3 claims, 2 of them naming Duplicate content; its own words name Duplicate content
Put exactly one absolute rel=canonical in the server-rendered head of every indexable page
Rests on 8 claims, 1 of them naming Duplicate content; its own words name Duplicate content
Differentiate same-language country versions beyond boilerplate and mark them with region codes
Rests on 6 claims, 1 of them naming Duplicate content; its own words name Duplicate content
Plan releases and fixes around Google's processing times, and judge results only after them
Rests on 11 claims, 1 of them naming Duplicate content; its own words name Duplicate content
Compute the expected URL for every request and redirect variants to it
Rests on 8 claims, 1 of them naming Duplicate content; its own words name Duplicate content
Also inglossary terms Duplicate cluster, Alternate names, Canonical, Similar URLs
Things named in the same claim, with the number of claims they share.