"algo-ecom-bm25"

BM25 Ranking Function

Overview

BM25 (Best Matching 25) is an improved TF-IDF ranking function that adds term frequency saturation and document length normalization. Score = Σ IDF(t) × (TF(t,d) × (k₁+1)) / (TF(t,d) + k₁ × (1 - b + b × |d|/avgdl)). Standard parameters: k₁=1.2, b=0.75. The backbone of most text search engines (Elasticsearch, Solr).

When to Use

Trigger conditions:

• Building product search with text-based relevance ranking
• Replacing basic TF-IDF with better document length normalization
• Tuning search relevance in Elasticsearch/Solr

When NOT to use:

• When semantic similarity matters more than keyword matching (use embeddings)
• For single-field exact matching (simpler methods suffice)

Algorithm

IRON LAW: BM25 Has Two Critical Parameters — k₁ and b
k₁ controls term frequency saturation: higher k₁ = more weight to
repeated terms. k₁=0 ignores TF entirely (boolean).
b controls document length normalization: b=1 fully normalizes by
length, b=0 ignores length. Default k₁=1.2, b=0.75 works for most
cases but MUST be tuned for your specific corpus.

Phase 1: Input Validation + Tokenization

Tokenize each document to lowercase word tokens. Remove stop words before counting — the bundled script drops a standard English stop list (the, a, an, and, or, but, of, in, on, at, to, for, with, by, from, as, is, are, was, were, be, been, being). Then build an inverted index: term → list of (document, term frequency). Compute: document lengths (post stop-word removal), average document length, document frequency per term.

⚠️ Stop-word removal affects |d| and avgdl: because stop words are dropped before length is measured, hand-computing BM25 without removing them will give the wrong length normalization and scores will be off by 3–5%. If you're reproducing BM25 by hand to compare against the script, apply the same stop list first — or just run the script.

Gate: Index built, statistics computed, corpus non-empty.