Phrase-based indexing is a family of Google patents by Anna Lynn Patterson, with a 2004 priority date, describing an index built from meaningful phrases rather than single words. Phrases are judged good if they predict the presence of other phrases, and documents are ranked and checked for spam by the related phrases they contain. Google has not confirmed production use. For a site, the practical reading is that topical depth shows up as the related phrases a subject naturally involves, and that stuffing them is a documented spam pattern.
What is phrase-based indexing?
Phrase-based indexing is the approach described in US7536408B2, Phrase-based indexing in an information retrieval system, assigned to Google with Anna Lynn Patterson as the inventor. It has a priority date of July 26, 2004 and was granted on May 19, 2009, per Google Patents. The same filing cross-references related applications on phrase identification, phrase-based searching, personalisation, taxonomy generation, document descriptions and duplicate detection.
Instead of indexing single words, the system indexes phrases, and it identifies which phrases are related to which. The IR idea behind it is co-occurrence: phrases that appear together more often than chance tell you what a document is about.
Phrases are identified that predict the presence of other phrases in documents.
Google (patent), US7536408B2: Phrase-based indexing in an information retrieval system, Google Patents
How does phrase-based indexing work?
The patent first separates good phrases from sequences of words that merely occur. It then measures how strongly each good phrase predicts others, using the ratio of actual to expected co-occurrence, which the patent calls information gain. Phrases with high information gain relative to one another become related phrases.
Documents are indexed by the phrases they contain and the related phrases present alongside them. At query time the system uses phrases in the query to retrieve and rank documents, and it uses the phrase data to cluster results, write document descriptions and remove duplicates.
a predictive measure is used which expresses the increased likelihood of one phrase appearing in a document given the presence of another phrase.
Google (patent), US7536408B2: Phrase-based indexing in an information retrieval system, Google Patents
How does phrase-based indexing detect spam?
A companion patent, US7603345B2, Detecting spam documents in a phrase based information retrieval system, granted October 13, 2009, uses the count of related phrases as a spam test. A normal document on a topic contains a modest number of related phrases; a document stuffed with them, typically generated to rank, contains far more. The patent gives spam documents as on the order of 100 to 1000 related phrases.
a normal, non-spam document will generally have a relatively limited number of related phrases, typically on the order of between 8 and 20, depending on the document collection.
Google (patent), US7603345B2: Detecting spam documents in a phrase based information retrieval system, Google Patents
What evidence is there that Google uses phrase-based indexing?
The evidence label is patent only. The family is large and was continued for years, but Google has not said that phrase-based indexing, or its spam test, runs in production. Modern Google systems such as BERT and neural matching model meaning in other ways, so even if the approach was used, its current role is unknown.
Claims that specific tools reproduce Google's related phrase lists are not supported by the patents.
What does phrase-based indexing mean for my site?
Cover a topic the way an expert would, and the related phrases follow. A page about a subject that never uses the phrases that subject naturally involves reads as shallow to readers and to any co-occurrence based system. Forcing in long lists of related phrases is the opposite failure and matches the spam pattern the patent describes.
A falsifiable test: extract the multi-word phrases that recur across the pages currently ranking for a query, then check which of them your page lacks. Missing phrases that represent real subtopics are coverage gaps; phrases you could only add by listing them are not.
Which Laurelin audit checks test for phrase-based indexing?
The new check, topic page omits the related phrases that recur across top-ranking pages, measures coverage of co-occurring phrases. It is a proxy for the patented method, not a reading of any Google index.
Related existing checks:
- Keyword stuffing and over-optimisation: the related-phrase stuffing the spam patent targets.
- Weak topic cluster structure: related phrases often belong on supporting pages, not all on one.
- Near-duplicate spun variants: the patent family includes duplicate detection by phrase.
- Commodity content: matching the phrase set without adding anything is not enough.
What are the key dates for phrase-based indexing?
Frequently asked questions about phrase-based indexing
Who invented phrase-based indexing at Google?
Anna Lynn Patterson is the named inventor on US7536408B2 and the related spam detection patent US7603345B2, both with a July 26, 2004 priority date.
Does Google use related phrases to detect spam?
A Google patent describes flagging documents with an unusually high number of related phrases as spam. Google has not confirmed the method is used.