Duplicate content

Day 1: Crawling 13

Said on stage 10

What Google's documentation says 1

What the press reported 1

Analysis by the author 1

Day 2: Indexing 48

Shown on screen 16

Said on stage 21

What Google's documentation says 7

Analysis by the author 4

Day 3: Serving: Ranking, Search Console, and Performance 2

Shown on screen 1

Said on stage 1

Across days and sessions 32

1. Stage D2-C429 Day 2 · Lightning session E: Managing Duplicates and Site Moves

   A community speaker said pages carry different link equity, and a canonical leader that is not the strongest page in its group is technically valid but most likely suboptimal for ranking, so the strongest page should be the leader.

   contradicts
   Slide D2-C348 Day 2 · Handling web duplication

   Google forwards the signals attached to every URL in a duplicate cluster, such as links, to the representative URL, so that nothing is lost by showing only one URL.

2. Stage D1-C379 Day 1 · How Google thinks about crawl budget

   The useless content worth improving or removing to save crawl budget includes very bad or low-quality pages, spam content, duplicate pages and soft error pages.

   extends
   Slide D1-C103 Day 1 · How Google thinks about crawl budget

   Four ways to manage crawl budget: use HTTP cache control, have good site navigation, restrict crawlers' access to faceted navigation and action URLs, and improve or remove useless content.

3. Stage D1-C444 Day 1 · Q&A

   Log files are worth checking on sites with filter parameters or complicated URLs, because crawlers can easily wander off into URLs that make no sense for the site.

   extends
   Stage D1-C397 Day 1 · Lightning session C: Crawling

   A community speaker said similar URLs are an indicator of duplicate content, broken links, unnecessary redirects and other technical issues.

4. Stage D1-C541 Day 1 · Q&A

   A Google panelist said, as a fun fact, that JavaScript was used for the language-consolidation redirects in Google's own site migration, because it was the only option available to the person doing it (the recording does not make fully clear whether the JavaScript performed the redirects or built the mapping).

   extends
   Stage D1-C479 Day 1 · Q&A

   Where content was duplicated across languages, Google's own site consolidation redirected two language versions into one, giving both old URLs one target path to redirect to.

5. Slide D2-C031 Day 2 · How is HTML interpreted

   Google extracts the rel=canonical link, through which site owners state their preferred canonical URL, and uses it in deduplication and in canonical selection.

   extends
   Docs D1-C128 Day 1 · session not recorded

   Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

6. Stage D2-C344 Day 2 · Handling web duplication

   Google deduplicates pages because many sites have very many pages and Google's index does not have room for everything.

   extends
   Slide D1-C037 Day 1 · How Search works and where's AI?

   For classic Search, crawling means Googlebot, scheduling and robots.txt, with AI used in parts such as scheduling. Indexing is one big but not limitless index that calculates signals and understands more than words, with AI such as BERT. Serving uses hundreds of signals tailored to the moment, with AI such as RankBrain.

7. Slide D2-C345 Day 2 · Handling web duplication

   Google's deduplication has three steps: identify and cluster duplicate web pages, pick representative URLs and index the unique pages, and forward signals to the representative URLs.

   extends
   Docs D1-C128 Day 1 · session not recorded

   Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

8. Slide D2-C345 Day 2 · Handling web duplication

   Google's deduplication has three steps: identify and cluster duplicate web pages, pick representative URLs and index the unique pages, and forward signals to the representative URLs.

   extends
   Stage D1-C207 Day 1 · How Search works and where's AI?

   Google clusters duplicate pages and selects one page per cluster as its representative, the canonical URL, so that users are not shown duplicates.

9. Stage D2-C346 Day 2 · Handling web duplication

   For deduplication, a cluster is a set of pages Google considers essentially equivalent: Google stores one of them in the index and keeps track of the other related URLs.

   extends
   Stage D1-C207 Day 1 · How Search works and where's AI?

   Google clusters duplicate pages and selects one page per cluster as its representative, the canonical URL, so that users are not shown duplicates.

10. Slide D2-C348 Day 2 · Handling web duplication

    Google forwards the signals attached to every URL in a duplicate cluster, such as links, to the representative URL, so that nothing is lost by showing only one URL.

    extends
    Stage D1-C111 Day 1 · session not recorded

    Gary Illyes said there is no such thing as a duplicate content penalty.

11. Stage D2-C367 Day 2 · Handling web duplication

    Soft 404 pages are another reason Google clusters pages together, an outcome site owners usually do not want.

    extends
    Docs D1-C073 Day 1 · How crawling errors affect Search

    A soft 404 is a page that returns a success code while its content looks like an error or an empty page. It is kept out of the index but continues to be crawled, wasting crawl budget.

12. Slide D2-C369 Day 2 · Handling web duplication

    Google's slide on structurally similar content asked whether a new URL that fits a known duplicate pattern, such as /buy/seo-service, even needs to be crawled.

    extends
    Slide D1-C094 Day 1 · How Google thinks about crawl budget

    If the quality or popularity of a URL is unknown, the aggregate quality or popularity of its parent path is used, then that path's parent, and so on.

13. Slide D2-C369 Day 2 · Handling web duplication

    Google's slide on structurally similar content asked whether a new URL that fits a known duplicate pattern, such as /buy/seo-service, even needs to be crawled.

    extends
    Docs D1-C128 Day 1 · session not recorded

    Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

14. Stage D2-C374 Day 2 · Handling web duplication

    A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.

    extends
    Stage D1-C069 Day 1 · How crawling errors affect Search

    DNS issues and firewalls are blind spots: Google does not know a firewall is blocking it, it only sees that requests fail.

15. Stage D2-C374 Day 2 · Handling web duplication

    A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.

    extends
    Analysis D1-C078 Day 1 · How crawling errors affect Search

    For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.

16. Stage D2-C374 Day 2 · Handling web duplication

    A growing cause of wrong clustering is bot protection: when a CDN or another protection system shows Googlebot the same 'you look like a bot, solve this puzzle' page on many unrelated URLs or domains, Google can cluster those pages as duplicates.

    extends
    Stage D1-C367 Day 1 · How crawling errors affect Search

    CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.

17. Stage D2-C375 Day 2 · Handling web duplication

    Google's systems find it very hard to recognise a bot-challenge page as an error page, and Googlebot cannot solve the puzzles, such as selecting all the cats, that those pages ask for.

    extends
    Stage D1-C367 Day 1 · How crawling errors affect Search

    CDN captcha challenges often return HTTP 200; Googlebot does not solve them and sees only content with a 200 status, which indexing then classifies as a soft 404, reported as an error in Search Console.

18. Docs D2-C376 Day 2 · Handling web duplication

    Google's Search Central post on CDNs says that if Google cannot recognise an error message served with a 200 status as an error, all pages showing the same message may be dropped from the index as duplicates, and recovery can be slow because Google has little incentive to recrawl duplicates. For bot-verification interstitials it recommends sending crawlers a 503 status.

    extends
    Analysis D1-C078 Day 1 · How crawling errors affect Search

    For temporary blocks return 503 or 429, never a 200 page with a captcha or error message, which becomes a soft 404 and can replace real content in the index. Verify Googlebot by reverse DNS, not by user agent.

19. Stage D2-C379 Day 2 · Handling web duplication

    rel=canonical is probably the most common way site owners signal duplicates, but it is very often wrong, for example a tag whose value reads 'canonical target' instead of a real URL (the example is partly unclear in the recording), so Google can only sometimes trust it.

    extends
    Slide D2-C031 Day 2 · How is HTML interpreted

    Google extracts the rel=canonical link, through which site owners state their preferred canonical URL, and uses it in deduplication and in canonical selection.

20. Stage D2-C380 Day 2 · Handling web duplication

    Many SEOs assume Google must follow their rel=canonical, but because people sometimes get it wrong, Google has to make its own judgment about the canonical.

    extends
    Analysis D1-C113 Day 1 · session not recorded

    The real cost of duplicates is control and crawling: Google may choose a canonical you did not want, and every copy is still crawled. Copied or scraped content is a separate spam-policy issue.

21. Slide D2-C382 Day 2 · Handling web duplication

    When localized pages are clustered, Google tries to use hreflang alternates; the speaker's short version of the advice was to use hreflang, which he called really helpful for same-language, different-country content.

    extends
    Slide D2-C033 Day 2 · How is HTML interpreted

    Google extracts hreflang annotations, through which site owners specify the language variants of their content, to know whether a page has an equivalent with similar content in another language.

22. Stage D2-C401 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    Broken canonical tags can make the wrong pages of a site show up in search results.

    extends
    Analysis D1-C113 Day 1 · session not recorded

    The real cost of duplicates is control and crawling: Google may choose a canonical you did not want, and every copy is still crawled. Copied or scraped content is a separate spam-policy issue.

23. Stage D2-C408 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    A community speaker advised auditing canonicals as whole canonical groups rather than pair by pair, paying special attention to each group's canonical leader.

    extends
    Docs D1-C128 Day 1 · session not recorded

    Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

24. Stage D2-C408 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    A community speaker advised auditing canonicals as whole canonical groups rather than pair by pair, paying special attention to each group's canonical leader.

    extends
    Stage D2-C398 Day 2 · Handling web duplication

    Google's speaker suggested checking rel=canonical links with a crawler such as Screaming Frog to make sure they are reasonable.

25. Stage D2-C444 Day 2 · Finding the gold nuggets: structured data, media, and more!

    Google runs deduplication before feature extraction, so that the expensive extraction work is spent only on a trimmed-down, manageable set of documents.

    extends
    Slide D2-C026 Day 2 · How is HTML interpreted

    A Google pipeline slide placed processing between the crawler and the index and listed six processing steps: HTML parsing, rendering, deduplication, feature extraction, signal extraction and index selection.

26. Stage D2-C701 Day 2 · Deciding what goes in the index?

    When Google already has duplicate information for a document, for example when reprocessing it, index selection uses it to drop non-canonical duplicates from further processing, so that, as the speaker put it, only canonicals end up in search results (a simplification; see D2-C703).

    extends
    Docs D1-C128 Day 1 · session not recorded

    Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

27. Stage D2-C701 Day 2 · Deciding what goes in the index?

    When Google already has duplicate information for a document, for example when reprocessing it, index selection uses it to drop non-canonical duplicates from further processing, so that, as the speaker put it, only canonicals end up in search results (a simplification; see D2-C703).

    extends
    Slide D2-C345 Day 2 · Handling web duplication

    Google's deduplication has three steps: identify and cluster duplicate web pages, pick representative URLs and index the unique pages, and forward signals to the representative URLs.

28. Stage D2-C882 Day 2 · Lightning session E: Managing Duplicates and Site Moves

    In a community case study, an old URL got a 301 redirect only when a new page served the same intent; URLs with no same-intent match were removed with an error status instead of being redirected (the exact code is unclear in the recording).

    extends
    Slide D2-C396 Day 2 · Handling web duplication

    Google's closing suggestions on duplication: use redirects for site migrations, use HTTP result codes, don't block agents, check your rel=canonical links, use hreflang links to help Google localize, report weird canonicals in the forums, make secure pages that work, and keep canonical signals clear.

29. Stage D3-C642 Day 3 · How long does it take to..?

    Google treats a site move as a complex canonicalization process in which every signal of the old site is recalculated and moved to the new one, and every indexing process has to run.

    extends
    Stage D2-C351 Day 2 · Handling web duplication

    Google treats a site migration as deduplication across sites, in which the site owner says the old and the new domain are the same and that Google should pick the new domain, so deduplication also helps Google handle migrations.

30. Stage D1-C207 Day 1 · How Search works and where's AI?

    Google clusters duplicate pages and selects one page per cluster as its representative, the canonical URL, so that users are not shown duplicates.

    repeats
    Docs D1-C128 Day 1 · session not recorded

    Google's canonicalization guide says some duplicate content on a site is normal and not a violation of its spam policies; Google clusters duplicate pages, picks the most representative one as canonical and crawls the duplicates less often.

31. Stage D1-C445 Day 1 · Q&A

    One fix for parameter variants of a URL is to work out the normalised URL and redirect the variants to it: the redirects cost crawl budget at first, but leave the site with a clean slate.

    repeats
    Stage D1-C403 Day 1 · Lightning session C: Crawling

    A community speaker advised that a web application compute the expected URL for every request, for example with reverse routing from the page type and ID, and redirect or return an error page when the requested URL differs.

32. Stage D2-C680 Day 2 · Deciding what goes in the index?

    Google's index is immense but finite, so Google cannot index every URL it finds on a web with a practically infinite number of URLs.

    repeats
    Stage D2-C344 Day 2 · Handling web duplication

    Google deduplicates pages because many sites have very many pages and Google's index does not have room for everything.

Built on these claims 21

Developer requirements 21

• ShouldDocumentedDEV-CAN-08

  Make pages that share a URL pattern differ in their main content

  Rests on 6 claims, 6 of them in this topic

• ShouldDocumentedDEV-URL-09

  If facet URLs must be crawlable, use & separators, a fixed filter order and 404 for empty results

  Rests on 5 claims, 4 of them in this topic

• MustDocumentedDEV-CAN-01

  Use permanent server-side redirects (301 or 308) for moved URLs, straight to the final URL

  Rests on 10 claims, 3 of them in this topic

• ShouldDocumentedDEV-CAN-09

  Keep migration redirects in place long term and judge a move in Search Console, not with site: queries

  Rests on 14 claims, 3 of them in this topic

• ShouldDocumentedDEV-INT-08

  Differentiate same-language country versions beyond boilerplate and mark them with region codes

  Rests on 6 claims, 3 of them in this topic

• ShouldDocumentedDEV-SDA-10

  Put the same structured data on duplicate URLs as on the canonical

  Rests on 3 claims, 3 of them in this topic

• AvoidDocumentedDEV-SRV-02

  Never serve bot challenges or error pages to crawlers with a 200 status

  Rests on 7 claims, 3 of them in this topic

• ShouldDocumentedDEV-URL-11

  Compute the expected URL for every request and redirect variants to it

  Rests on 8 claims, 3 of them in this topic

• MustDocumentedDEV-CAN-02

  Serve one host over HTTPS with a valid certificate and redirect every other origin to it

  Rests on 4 claims, 2 of them in this topic

• MustDocumentedDEV-SRV-01

  Let verified Google crawlers through firewalls, CDNs and bot protection

  Rests on 11 claims, 2 of them in this topic

• ShouldDocumentedDEV-AIF-02

  Spend no effort on llms.txt, AI text files or AI-specific markup for Google

  Rests on 15 claims, 1 of them in this topic

• MustDocumentedDEV-CAN-03

  Put exactly one absolute rel=canonical in the server-rendered head of every indexable page

  Rests on 8 claims, 1 of them in this topic

• MustDocumentedDEV-CAN-05

  Make all canonical signals agree: redirects, rel=canonical, internal links, sitemaps and hreflang

  Rests on 6 claims, 1 of them in this topic

• AvoidDocumentedDEV-CAN-07

  Never leave staging, preview or mirror copies of the site publicly crawlable

  Rests on 2 claims, 1 of them in this topic

• MustDocumentedDEV-CAN-10

  Redirect an old URL only to a page with the same intent, and return 404 or 410 when there is none

  Rests on 7 claims, 1 of them in this topic

• MustDocumentedDEV-ERR-01

  Return 404 or 410 for removed and non-existent URLs

  Rests on 7 claims, 1 of them in this topic

• AvoidDocumentedDEV-ERR-03

  Never serve error states, empty results or failed data loads with a 200 status

  Rests on 10 claims, 1 of them in this topic

• ShouldDocumentedDEV-HTM-01

  Wrap each page's primary content in one clearly delimited main area

  Rests on 14 claims, 1 of them in this topic

• AvoidDocumentedDEV-INT-02

  Do not redirect visitors automatically by IP address or browser language

  Rests on 2 claims, 1 of them in this topic

• ShouldDocumentedDEV-INT-07

  Make each page's language unambiguous by translating the template with the main content

  Rests on 12 claims, 1 of them in this topic

• ShouldDocumentedDEV-MON-06

  Audit canonical groups with a crawler on a schedule

  Rests on 5 claims, 1 of them in this topic

Sources 21

• In-depth guide to how Google Search works Google Search Central · checked 3 October 2026
• What is canonicalization Google Search Central · checked 3 October 2026
• Optimizing your website for generative AI features on Google Search Google Search Central · checked 3 October 2026
• What Crawl Budget Means for Googlebot Search Central blog (16 January 2017) · checked 3 October 2026
• Optimize your crawl budget Google · checked 4 October 2026
• URL structure best practices for Google Search Google Search Central · checked 3 October 2026
• How to specify a canonical URL with rel="canonical" and other methods Google Search Central · checked 3 October 2026
• How to move a site Google Search Central · checked 4 October 2026
• Demystifying the "duplicate content penalty" Search Central blog (12 September 2008) · checked 2 October 2026
• Google Says It Can Handle Multiple URLs To The Same Content Press Search Engine Journal (8 April 2026) · checked 2 October 2026
• A guide to Google Search ranking systems Google Search Central · checked 3 October 2026
• Redirects and Google Search Google Search Central · checked 4 October 2026
• URL Inspection tool Google Search Console Help · checked 4 October 2026
• Tell Google about localized versions of your page Google Search Central · checked 3 October 2026
• Fix canonicalization issues Google Search Central · checked 3 October 2026
• Crawling December: CDNs and crawling Search Central blog (24 December 2024) · checked 4 October 2026
• Troubleshoot Google Search crawling errors Google Search Central · checked 2 October 2026
• Managing multi-regional and multilingual sites Google Search Central · checked 3 October 2026
• General structured data guidelines Google Search Central · checked 3 October 2026
• Page indexing report Google Search Console Help · checked 4 October 2026
• Spam policies for Google web search Google Search Central · checked 4 October 2026