How Search Engines Process Real-Time Indexing

A client of mine — a mid-sized e-commerce operator with 3,400 product pages — ran a routine Search Console audit in April 2026. Of those 3,400 URLs, 2,980 sat at “Discovered — currently not indexed.” Not blocked. Not noindexed. Discovered. Meaning Google had seen them, decided they weren’t worth processing yet, and moved on.

That distinction — between crawled and indexed — is where most SEO advice falls apart. People optimize for discovery when the real fight happens 30 milliseconds after Googlebot leaves.

This piece is about what happens in those 30 milliseconds, and the minutes, hours, and sometimes weeks that follow. Not the marketing version. The operational version — including what the 2024 API leak actually revealed about Google’s internal index architecture that Google itself never disclosed.

“Real-Time Indexing” Is a Partial Truth

Google has been calling its system real-time since the Caffeine rollout in 2010, when the company said the update delivered 50% fresher results compared to the previous layered index. That claim is technically accurate and operationally misleading at the same time.

Caffeine is real-time in the sense that pages are analyzed as they’re crawled — there’s no batch-processing delay between “Googlebot fetched this page” and “Google starts evaluating it.” But real-time analysis is not the same as real-time indexing. What determines whether your page gets indexed today, next week, or never isn’t the analysis speed. It’s the crawl priority assigned before Googlebot even knocks on your server.

The practical implication: if you publish content and it doesn’t appear within 48–72 hours, the problem usually isn’t analysis speed. It’s that your page didn’t make it far enough up the priority queue to get fetched in the first place.

The Crawl Priority Engine — What Actually Runs It

Google’s crawl scheduler isn’t documented publicly, but from a combination of the 2024 API leak analysis by Mike King at iPullRank, patent filings, and empirical behavior, the priority system operates on roughly four input classes:

1. PageRank (not the toolbar version — the live one)

The leak confirmed seven variants of PageRank in active use. Pages with stronger internal link equity from already-indexed, high-authority pages get crawled more often and more quickly. This is why a new post on a domain with strong existing content can index in minutes, while an identical post on a new domain can wait weeks.

2. Freshness signals

Documents that have changed recently — measured by HTTP headers, content hashes, and sitemap lastmod values — receive elevated priority. The system learns your update patterns: if your news section historically publishes every six hours, Googlebot allocates more crawl bandwidth to that path.

3. Crawl budget allocation

Each site receives a crawl budget — effectively a rate limit on how many pages Googlebot will fetch per unit of time. Faceted navigation is responsible for roughly 50% of crawl budget issues observed in 2025–2026 audits, because parameter-driven URLs multiply the apparent page count exponentially without adding indexable value.

4. Server response quality

Response time directly affects crawl rate. A server that responds in 200ms gets crawled more aggressively than one that takes 2.2 seconds. Google self-throttles to avoid overloading slow servers, which inadvertently rewards fast infrastructure with more frequent visits.

The Rendering Gap: What Googlebot Actually Sees

After Googlebot fetches a page, it doesn’t immediately process it. The raw HTML goes into a rendering queue — Google’s Web Rendering Service (WRS), which runs a headless version of Chrome. This is where JavaScript-heavy sites create a structural disadvantage that doesn’t show up in any standard SEO audit.

The fetch and the render happen at different times. Sometimes minutes apart. Sometimes hours. In 2025–2026, improvements brought WRS closer to the latest Chrome versions, but the delay still exists — and any content dependent on JavaScript that fires after the initial page load risks being invisible to the indexer.

The practical test: request your URL via Google Search Console’s URL Inspection tool and examine the rendered HTML. If what you see in “View Tested Page” doesn’t match what a human sees in a browser, you have a rendering gap.

The Three Tiers Nobody Told You About (Until They Leaked)

This is the part that reframes everything. Until May 2024, Google officially denied having anything resembling a domain authority score. It also never publicly documented how content was stored and prioritized within the index itself. Then the API documentation leaked.

What the 2024 leak revealed — first surfaced by Rand Fishkin and decoded by Mike King of iPullRank — is that Google’s index is stratified into three named tiers managed by a system called SegIndexer:

The attribute governing placement is called scaledSelectionTierRank — a language-normalized score that places each document in its fractional position within the quality hierarchy. What’s critical to understand: this tier assignment isn’t just about where your page is stored. It affects link equity. According to the leaked AnchorsAnchor module documentation, links from Base-tier pages carry significantly more weight than links from Landfill-tier pages. The link economy is tiered too.

What puts you in Landfills? The Panda system — still very much active — creates what the docs call “long-term algorithmic debt.” Signals like pandaDemotion, babyPandaDemotion, and low vlq (very low quality) scores persist. They don’t reset on the next crawl. A site that published thin content in 2023 and cleaned it up in 2025 may still be carrying Panda debt into 2026.

What Caffeine Actually Does (and Doesn’t Do)

Caffeine is the indexing infrastructure layer, not an algorithm. Think of it as the factory floor — it determines how fast raw content moves from “fetched” to “stored in the index.” It’s been in production since 2010 and has been incrementally improved since.

Its core innovation was replacing Google’s previous batch-update model — where the entire index was refreshed periodically in large chunks — with a continuous, incremental system. New content is analyzed and incorporated into the serving index as soon as it’s processed, without waiting for a full cycle to complete.

What Caffeine doesn’t control: which tier your content lands in, how often your URLs get crawled, or whether they make it to the crawl queue at all. Those are separate systems with their own inputs.

AI Changes the Crawl Pipeline — But Not How Most People Think

Since 2025, Googlebot has operated alongside a growing fleet of AI-specific crawlers. As of mid-2026, Google’s AI Mode — the generative search interface — surfaces “preferred sources” in AI Overviews and AI Mode, each clearly labelled, alongside traditional SERP results. Getting selected as a preferred source and getting ranked in the top-10 are increasingly divergent optimization targets.

Three things have changed in the crawl pipeline specifically because of AI:

Context-aware rendering

Google’s crawler now adapts its rendering pipeline based on content type, using MUM (Multitask Unified Model) to understand relationships between content formats. A page with tightly structured, machine-readable information is processed differently from one with unstructured narrative text.

The llms.txt signal

A new 2025–2026 standard worth implementing: the llms.txt file in your site root — effectively a robots.txt equivalent structured for language model consumption. It provides LLMs with a concise, structured description of your site and its key content in a format they parse far more efficiently than standard HTML. It’s not a ranking signal yet. It’s infrastructure for what’s coming.

Google-Extended opt-out

Googlebot’s crawling now supports both traditional indexing and AI training data collection. Sites that want to opt out of training data specifically — while remaining indexed — can use Google-Extended in their robots.txt. This is distinct from blocking Googlebot entirely.

The Information Gain Problem — Why Good Content Still Doesn’t Index

Here’s the failure mode I haven’t seen explained accurately anywhere: a page can be technically perfect and still fail to index because of what the system internally calls “Information Gain.”

In the 2025–2026 period, a regional portal with 750 server-side rendered pages saw 96% of its URLs fail to index. The site was technically clean — fast, properly structured, no crawl blocks. The 342:1 ratio of “Discovered” to “Crawled” URLs pointed to a single cause: insufficient Information Gain. Google determined that most of those pages added nothing to what was already indexed.

Information Gain isn’t just about uniqueness. It’s about whether your content changes the user’s understanding of a topic in a way that Google doesn’t already have covered elsewhere in its index. Aggregated, summarized, or slightly-reworded content fails this test even when it passes plagiarism checks. The question isn’t “is this original?” but “does the index need this?”

What You Can Actually Control

Given everything above, here’s where intervention is meaningful versus where it isn’t.

Quick-audit checklist before your next publish

• Server-side rendered HTML contains all critical content (H1, meta description, main body text, internal links)
• New URL has at least one contextual internal link from an already-indexed, linked-to page
• URL is included in sitemap with accurate lastmod
• Page adds something the existing top-3 results don’t have
• No parameter variations of this URL are being generated (check faceted nav / sorting / filtering)
• Do not use “Request Indexing” for more than ~10 URLs per day — GSC rate-limits this and overuse reduces its effectiveness
• Do not update lastmod without making a meaningful content change — Google tracks the accuracy of this signal

The Part Nobody Has Mapped Yet

There’s a question I genuinely don’t have a clean answer to: how does a page move between tiers after initial placement?

The leak documents suggest tier rank is recalculated — the scaledSelectionTierRank isn’t set once and frozen. But the recalculation frequency and the specific signals that trigger upward movement are opaque. What’s known from empirical behavior: pages that consistently attract clicks, generate dwell time, and earn fresh backlinks appear to migrate upward. Pages that stagnate — same content, declining engagement, no new inbound links — appear to drift toward lower tiers over time.

Whether this is a continuous gradient or a periodic batch re-score, nobody outside Google knows with certainty. Including me. If you have Search Console data showing tier-movement patterns, I’d genuinely be interested — it’s one of the gaps in current public knowledge that matters most for technical SEO strategy.