Deduplication systems are Google's confirmed ranking systems that stop near identical pages from filling the results. When several pages are very similar, Google shows the most relevant one, and since January 2020 a page shown as a featured snippet is not repeated on the first page. Canonicalisation is the related indexing step that picks one URL from a set of duplicates. For a site, duplicate URLs compete with each other and only one is likely to be shown, so consolidate them and signal a clear canonical.
What are Google's deduplication systems?
Deduplication systems are the parts of Google Search that stop very similar pages from appearing repeatedly for the same search. Google lists them in its guide to ranking systems: when many matching pages are near copies, Google shows only the most relevant results.
The same guide says deduplication also applies to featured snippets. Danny Sullivan confirmed on 22 January 2020 that a page elevated to a featured snippet is no longer repeated in the first page of results (Search Engine Roundtable).
Searches on Google may find thousands or even millions of matching web pages. Some of these may be very similar to each other. In such cases, our systems show only the most relevant results to avoid unhelpful duplication.
Google, A guide to Google Search ranking systems, Google Search Central
How does deduplication work?
There are two layers. At indexing, canonicalisation groups duplicate URLs and picks one representative URL to index; Google's documentation says this process is often called deduplication. At serving, ranking systems filter results that are very similar to others already shown, including the featured snippet case.
Canonicalisation uses signals such as redirects, rel=canonical, sitemaps and internal links, but Google makes the final choice. Search Console's Page indexing report exposes the outcome through statuses such as "Duplicate without user-selected canonical" and "Duplicate, Google chose different canonical than user".
- Indexing layer: duplicate URLs are clustered and one canonical is chosen.
- Serving layer: very similar results are filtered so one appears.
- Featured snippets: the snippet counts as the page's first page listing.
Often called deduplication, this process helps Google show only one version of the otherwise duplicate content in its search results.
What evidence is there that Google uses deduplication?
This page is labelled Confirmed. Deduplication systems are named in Google's ranking systems guide, canonicalisation is documented in Search Central, and the featured snippet change was confirmed on record by Google's Danny Sullivan in January 2020.
Google also states that some duplicate content on a site is normal and is not a spam violation (Google Search Central). The cost of duplication is usually filtering and diluted signals, not a penalty.
If a web page listing is elevated into the featured snippet position, we no longer repeat the listing in the search results.
Danny Sullivan, Google, Google Drops A Site's Normal Snippet When Featured Snippet Is Displayed, Search Engine Roundtable
What does deduplication mean for your site?
If two of your URLs carry near identical content, they compete and only one is likely to appear. The one Google picks may not be the one you want, and links and internal links split across both.
Consolidate where the pages serve the same purpose, using 301 redirects or a consistent rel=canonical, and make pages that must coexist, such as location or product variants, substantially different. A falsifiable test: after consolidation, the count of "Duplicate" statuses in the Page indexing report for that template should fall, and impressions should concentrate on the chosen URL.
Which Laurelin audit checks test for deduplication?
The new check Search Console reports Google choosing a different canonical than declared looks at the outcome of deduplication directly. It measures Google's reported decision, which is the closest observable output of the system.
Related checks: Multiple URLs serving identical content finds the cause. Internal duplicate content and Near duplicate spun variants find page level duplication. Missing canonical tag leaves the choice to Google. Non-canonical URL indexed shows the wrong version winning. External duplicate content covers copies on other domains.
What are the key dates for Google deduplication?
- 2020-01-22: Featured snippet pages no longer repeated on the first page, confirmed by Danny Sullivan (source)
Frequently asked questions about Google deduplication
Is duplicate content a Google penalty?
No. Google says some duplicate content is normal and not a spam violation. The usual effect is that only one version is shown and signals are split.
Does a featured snippet count as a first page ranking?
Yes. Since January 2020 Google does not repeat a featured snippet page as a separate listing on the first page, and the snippet counts as one of the listings.
How do I see which URL Google chose?
Use the URL Inspection tool and the Page indexing report in Search Console, which show the Google selected canonical and duplicate statuses.