--- type: "Article" title: "Google Caffeine | technical SEO foundations" description: "What Google's Caffeine indexing system is, how Percolator made incremental indexing possible, and how to test how fast your new pages get indexed." resource: "https://www.laurelinlabs.com/search-systems/caffeine" tags: ["google-caffeine", "technical-seo-foundations", "percolator-incremental-indexing"] generated: { by: "laurelin-site-build/2026-09-02", at: "2026-09-02T12:00:00+00:00" } status: "stable" stale_after: "2027-03-02T00:00:00+00:00" --- # Caffeine Caffeine is the web indexing system Google completed in June 2010. It replaced a layered, batch-refreshed index with one that updates continuously as pages are crawled, and Google said it delivered 50 percent fresher results. A Google paper presented at OSDI 2010 describes the Percolator system underneath it. Caffeine is infrastructure, not a ranking signal. For a site, it means new and changed pages can reach the index quickly, provided Google can discover and crawl them. ## What is Caffeine? Caffeine is the web indexing system Google announced as complete on [8 June 2010](https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html). Google said it provided [50 percent fresher results](https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html) for web searches than its previous index and was the largest collection of web content Google had offered. It changed how the index is built, not how results are ranked. It is best understood as the reason freshness became possible at scale. ## How did Caffeine change Google's indexing? Google's old index had layers refreshed at different rates, and the main layer updated [every couple of weeks](https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html), because refreshing meant reprocessing the whole web. Caffeine instead analyses the web in small portions and adds new or changed pages to the index as they are found. The engineering is described in a Google paper by Daniel Peng and Frank Dabek presented at [OSDI 2010](https://research.google/pubs/large-scale-incremental-processing-using-distributed-transactions-and-notifications/), the USENIX Symposium on Operating Systems Design and Implementation. It introduces Percolator, a system for incremental processing built on Bigtable, and says the Percolator-based indexing system is known as Caffeine. The paper reports that the [median document moves through Caffeine over 100x faster](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Peng.pdf) than the previous system and that its document collection was 3x larger. ## What evidence is there that Google uses Caffeine? The evidence label is Confirmed. Google announced Caffeine on its official blog and its engineers published the underlying system in a peer-reviewed paper. The paper's abstract states that Percolator was deployed to create the Google web search index. What is not confirmed is how Caffeine has changed since 2010. None of the sources on this page describe the current architecture, so any claim about it beyond continuous, incremental indexing should be treated as unverified. ## What does Caffeine mean for a site? Continuous indexing removes the index as the bottleneck. The remaining delay between publishing and being indexed is mostly discovery and crawling, which a site controls: internal links from frequently crawled pages, accurate sitemaps and a healthy server. A falsifiable way to measure it: record publish time for new URLs, then the first Googlebot request in server logs and the date Search Console URL Inspection reports the page as indexed. If comparable sites index new pages within hours and yours takes days, the gap is in discovery or crawl priority, not in Google's index. ## Which Laurelin audit checks test for Caffeine? The new check [New URLs take days to be crawled and indexed after publication](https://www.laurelinlabs.com/audit/technical-foundations/361-new-urls-take-days-to-be-crawled-and-indexed-after-publication) measures publish-to-index latency. Caffeine is Google infrastructure and cannot be tested from a site; this check measures a proxy, the part of the pipeline a site controls. Related existing checks: [Page only in sitemap, not internally linked](https://www.laurelinlabs.com/audit/technical-foundations/38-page-only-in-sitemap-not-internally-linked): pages without internal links are discovered slowly. [Excessive crawl depth](https://www.laurelinlabs.com/audit/technical-foundations/39-excessive-crawl-depth): deep pages are crawled less often. [lastmod missing or always now](https://www.laurelinlabs.com/audit/technical-foundations/107-lastmod-missing-or-always-now): unreliable lastmod removes a recrawl hint. [Sitemap not submitted to Search Console](https://www.laurelinlabs.com/audit/technical-foundations/102-sitemap-not-submitted-to-search-console): a missing discovery route. [Excessive TTFB](https://www.laurelinlabs.com/audit/technical-foundations/80-excessive-ttfb): slow responses can reduce how much Google crawls. ## What are the key dates for Google Caffeine? - 2010-06-08: Google announces completion of the Caffeine indexing system ([source](https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html)) - 2011-11-03: Google freshness update builds on Caffeine ([source](https://googleblog.blogspot.com/2011/11/giving-you-fresher-more-recent-search.html)) ## FAQ ### Is Caffeine a ranking algorithm? No. Caffeine is Google's indexing system. It changed how quickly pages enter the index, not how they are ranked. ### What is Percolator? Percolator is the incremental processing system described in a 2010 Google paper by Daniel Peng and Frank Dabek. The paper says the Percolator-based indexing system is known as Caffeine. Hub: [Technical Foundations](/topics/technical-foundations.md)