How Search Engines Rank You (Roughly)

I set up a self-hosted search engine over a weekend a while back, expecting the ranking logic to be the hard part. It wasn’t. Building an index that could find every document containing a given word took an afternoon. Deciding which of those documents to show first, in an order a human would actually agree with, is the part search companies have spent billions of dollars and twenty-five years refining, and it’s also the part that most explains why SEO advice sounds so contradictory: different search engines, and different eras of the same search engine, weigh the underlying signals differently, but almost all of them are still built on the same two ideas from the 1970s and 1990s, dressed up with a great deal of additional machinery on top.

The inverted index is the part everyone gets right

Before you can rank anything, you need to find every document that mentions a word, quickly, across a corpus that might contain billions of pages. Scanning every document’s text at query time obviously doesn’t scale, so search engines invert the problem: instead of a list of documents each containing a list of words, they build a list of words, each pointing at the list of documents containing it. That’s the inverted index, and it’s genuinely one of the older ideas in computer science still doing load-bearing work in production systems today.

1
2
3
4
5
6
7

{
  "index": {
    "server":  [12, 45, 189, 204],
    "cheap":   [12, 88, 204],
    "hosting": [12, 45, 88, 204, 501]
  }
}

Querying “cheap server hosting” against that structure means intersecting the document lists for all three terms — document 12 and document 204 appear in every list, so they’re candidates; document 45 is missing “cheap” and drops out. This intersection is fast precisely because it never touches the actual document text at query time, only the pre-built lists, which is why full-text search over millions of documents can return in milliseconds even on comparatively modest hardware — the expensive work (building the index) happened once, offline, ahead of any query arriving.

TF-IDF and BM25 answer “how relevant, roughly”

Finding candidate documents is only half the problem — you also need to rank the candidates that matched against each other, and this is where term frequency–inverse document frequency comes in, an idea from 1970s information retrieval that’s still the conceptual backbone of relevance scoring today. The intuition is straightforward: a word that appears often in one specific document, but rarely across the whole corpus, is a strong signal that the document is specifically about that word. A word that appears constantly in nearly every document — “the,” “and,” “is” — carries almost no discriminating power no matter how many times it shows up in any single one, because its presence tells you nothing about what makes this particular document different from the rest.

1

TF-IDF(term, doc) = term_frequency(term, doc) × log(total_docs / docs_containing(term))

BM25, the ranking function that actually underpins most modern search systems including Elasticsearch’s and OpenSearch’s defaults, refines this with two practical adjustments TF-IDF gets wrong on its own. First, it caps the benefit of repeating a term — a document that says “server” fifty times isn’t proportionally fifty times more about servers than one that says it once, and BM25’s saturation curve reflects that diminishing return rather than scoring raw frequency linearly. Second, it normalises for document length, because a genuinely relevant short document that mentions your search term twice deserves to outrank a padded, unfocused long document that mentions it twice purely by accident of having more words overall. Both adjustments come from decades of empirical tuning against real search-quality evaluations, and they’re why BM25 has remained the default relevance function in production search systems for so long despite plenty of fancier alternatives being tried against it.

PageRank answered a different question entirely

TF-IDF and BM25 only ever look at a document’s own text. They can’t distinguish an authoritative, well-regarded page from a low-effort page that happens to repeat the right keywords the right number of times, because word frequency alone carries no signal about quality or trust. Google’s original contribution wasn’t a better relevance score for text — it was recognising that the link structure of the web itself carries a separate, independent signal: a page linked to by many other pages, especially pages that are themselves heavily linked to, is more likely to be worth trusting than a page nobody links to at all, regardless of what its own text says about itself.

PageRank formalises this as a probability distribution: imagine a hypothetical user clicking links at random forever, occasionally jumping to a random page instead of following a link, and PageRank is the fraction of time that random walk spends on each page in the long run. A page accumulates rank by being linked to from other pages, and it passes on a fraction of its own rank through every link it makes outward, so a link from a page with high rank is worth considerably more than a link from a page with almost none — which is precisely the property that made link farms and reciprocal-linking schemes an obvious, gameable target the moment PageRank’s influence on rankings became public knowledge, and precisely the property every subsequent anti-spam effort in search has had to defend.

Modern ranking systems combine both signals and dozens more on top — click-through data, dwell time, freshness, mobile-friendliness, page speed, and increasingly machine-learned models trained directly on which results users actually clicked and stayed on — but the two-part foundation hasn’t gone anywhere: find the candidates with an inverted index, score their textual relevance with something in the BM25 family, then re-rank using authority signals descended conceptually from PageRank. Everything else is refinement layered on top of that same skeleton, which is also the reasoning behind what the algorithm actually is on your feed in a social-media context — a different set of signals, engagement rather than links, feeding the identical underlying pattern of candidate retrieval followed by learned re-ranking.

Semantic search is a third stage bolted on more recently

Everything above scores documents on the literal words they share with a query, which is precisely why keyword-based search fails on synonyms and paraphrase — a document about “affordable web hosting” gets no relevance credit at all for a query about “cheap server hosting” under pure BM25, because “affordable” and “cheap” are entirely different tokens to an inverted index, however obviously related they are to a human reader. Semantic or vector search addresses this by representing both documents and queries as embeddings — dense numeric vectors produced by a language model, positioned so that texts with similar meaning end up near each other in that vector space regardless of which literal words they used — and ranking candidates by vector similarity (typically cosine similarity) rather than, or alongside, token overlap.

In practice, production systems increasingly run a hybrid of both: keyword-based BM25 retrieval for candidates that share literal terms with the query, semantic vector retrieval for candidates that mean the same thing without sharing terms, and a final re-ranking stage that blends both scores before a result set is returned. This hybrid approach is exactly what underpins running embeddings locally for self-hosted search, and it’s worth building on top of a working BM25 index rather than instead of one, because vector search alone is notably weaker than BM25 at exact-match queries — an error message, a product SKU, a precise legal citation — where the literal string genuinely is the thing you’re looking for, and a purely semantic system will happily return conceptually related but factually wrong results in exactly the cases where precision matters most.

Why SEO advice contradicts itself

Once you see the two-stage structure — relevance scoring, then authority re-ranking — most of the contradictory SEO folklore resolves into “which stage is this advice actually targeting.” Keyword density advice targets the BM25 stage and is mostly obsolete now, because search engines caught up to keyword stuffing decades ago and BM25’s own frequency-saturation curve already blunts the benefit of repetition past a fairly low point. Backlink-building advice targets the PageRank-descended stage and remains genuinely, if now-heavily-policed, effective, because the underlying signal — do other trusted sites vouch for this one by linking to it — is still architecturally central to how modern engines separate a trustworthy result from a keyword-stuffed one. Content-freshness and engagement advice targets neither classical stage directly, but the machine-learned re-ranking layer that’s been bolted on top of both since roughly the mid-2010s, trained on click and dwell-time data the original two-stage model never had access to at all.

This is also why building your own search engine, whether for a homelab document collection or a small site, is worth doing with a self-hosted, privacy-respecting engine rather than assuming you need to reinvent Google’s entire stack — an inverted index plus BM25 alone gets you genuinely good relevance for a personal or small-scale corpus, because the PageRank-style authority signal exists specifically to solve trust-at-web-scale, a problem a small, curated document set generally doesn’t have in the first place.

Troubleshooting a self-hosted search index

The most common disappointment when standing up your own search stack is discovering that a naive TF-IDF implementation ranks short, low-quality documents above long, genuinely useful ones, purely because a short document’s few words got a disproportionate relevance boost from the raw term-frequency ratio. Switching to BM25’s length-normalised scoring (built into Elasticsearch, OpenSearch, Tantivy and most modern search libraries as the default rather than something you implement by hand) usually resolves this immediately, because the length-normalisation term specifically counteracts the bias.

The second common issue is stemming and tokenisation mismatches — a query for “hosting” that fails to match a document containing only “hosted” or “host,” because the index was built without a stemmer reducing every word to a common root before indexing. Nearly every production search library ships a configurable analyser chain (lowercase, stem, strip stop words) for exactly this reason, and it’s worth verifying explicitly which analyser is active, because the default sometimes differs from what a quick-start tutorial assumes.

The third is treating relevance score as the only signal worth tuning, and neglecting the equivalent of PageRank entirely — for a document collection with genuine internal cross-references (a wiki, a knowledge base, a set of interlinked articles), weighting documents by how many internal links point to them, the same way search engines weight external links, is a cheap, high-value addition most self-hosted search setups skip purely because it isn’t in the tutorial. A fourth, increasingly common issue once teams add an embeddings model on top, is discovering that vector similarity alone returns confidently wrong results for exact-match queries — a support ticket search that returns semantically related but wrong error codes because nothing in the vector representation privileges an exact string match the way a keyword index naturally does. The fix, again, is running both retrieval paths side by side rather than replacing one with the other, and only trusting a purely semantic result once a keyword search has confirmed there genuinely isn’t a literal match to prefer instead.

Is it worth understanding the mechanism

Yes, whether you’re optimising a site for search or building your own. The inverted index and BM25 are stable, well-documented, decades-old technology you can genuinely master and reason about precisely, and PageRank’s core insight — that structural authority is a separate axis from textual relevance — explains a surprising amount of both search-engine behaviour and, more broadly, how any ranked system that combines “does this match” with “is this trustworthy” tends to work. The learned re-ranking layers on top are genuinely opaque and shift constantly, but the foundation underneath them hasn’t changed shape in twenty-five years, and understanding it is worth far more than chasing whichever ranking-factor rumour is circulating this month.