Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarRaft→ original

In RAG, Classic BM25 Found the Answer in 72% of Queries; Neural Search Found It in 49.7%

When a RAG bot fails to find a case by its number or a person by surname, neural search is often to blame: it captures meaning but confuses rare words. On EntityQuestions, a test with questions about specific names, BM25 (classic keyword search) found the answer in 72% of cases, and neural search in 49.7%. So in production, keep BM25 alongside embeddings and blend their scores with a weighted sum.

AI-processed from Raft; edited by Hamidun News
In RAG, Classic BM25 Found the Answer in 72% of Queries; Neural Search Found It in 49.7%
Source: Raft. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceRaft ↗
Editor-in-Chief
Key metric #1BM25 scores from different libraries are not comparable: each has its own formula
Key metric #2Since version 8, Lucene has dropped the (k1+1) multiplier in the numerator of the formula
Key metric #3Tune thresholds within a single library and a single version

Imagine a RAG bot for legal documents: it first searches a database for relevant fragments, then uses them to answer the user. You ask about a specific case by its number, or about a person by surname, and vector (neural) search returns texts that are "similar in meaning" but not the ones you need. The classic BM25 algorithm, which searches by word matches and weights words by their rarity, handles such queries better.

On EntityQuestions, a test that asks about specific names and entities, BM25 found the right answer in 72% of cases, while neural search found it in only 49.7%. An engineering breakdown by an AI architect who built search over legal documents (the "NeuroYurist" project) shows how to treat BM25 as a data structure and what to do with it in production.

BM25 as a precomputed table

Imagine that every word in every document has a score assigned in advance: how important that word is for that text. Search then boils down to adding up the ready-made scores of the words in the query. That is why BM25 is so fast.

Technically, the weight of a "word–document" pair depends on the word's frequency in the document, the document's length, and global corpus counters, but not on the query. That means it can be computed at indexing time and stored in a sparse matrix. Search becomes a matter of selecting columns by the query tokens and summing them. The article includes a minimal Python example with numpy and scipy: about 20 lines, the IDF variant (word-rarity scores) from Lucene, and top results selected via argpartition.

This scheme has one downside, but an unpleasant one: scores depend on the entire collection. Add or remove a document, and formally the weights of all the others change. You either have to rebuild the matrix or knowingly live with statistics drift, for example in average document length. In a distributed system, sharding adds another problem: a shard's local statistics differ from the global ones, so scores from different nodes get blended incorrectly. Elasticsearch addresses this with the dfs_query_then_fetch mode at the cost of an extra network round trip, but if the corpus fits on a single node, it is simpler to keep one shard.

"BM25" is not a single formula

There is no single standard. Kamphuis et al. (ECIR 2020) compared eight common variants: Robertson's classic, Lucene, ATIRE, BM25L, BM25+ and others. They differ in how they compute IDF, how they normalize term frequency, and how they handle negative values (which appear when a word occurs in more than half of the documents).

Practical consequences:

  • The same documents get different numbers in different libraries, so comparing them across systems is meaningless.
  • Starting with version 8, Lucene removed the (k1+1) multiplier in the numerator. The order of documents within a query does not change, but absolute scores do, which breaks comparisons across versions.
  • A BM25 score is not a probability: it has no upper bound and grows with query length. A hard threshold such as "above 12 means relevant" will inevitably break. Use relative thresholds: a fraction of the maximum score, or a normalized value.
  • In sparse embedders that map words to indices via hashing, large vocabularies with case numbers and official details produce collisions: different words get merged into one coordinate. In such domains, it is better to switch to an explicit vocabulary right away.

What the benchmarks show and how to blend with a neural model

The numbers were given above: on EntityQuestions, BM25 found the right answer in 72% of cases and neural search in 49.7%. Neural models handle rare terms poorly (surnames, case numbers, part numbers) and "prefer" short fragments; on long texts they underperform.

For a hybrid, the author recommends blending BM25 and the neural model with a weighted sum of scores rather than by ranks (the RRF method, which only considers a document's position in the list). One caveat matters: if you have fine-tuned the embedder on your own data, the blend may get worse until the weights are re-tuned. So after every model change, recompute the weights rather than carrying them over from the old version.

Another takeaway: search quality depends most on text preparation, meaning morphology, tokenization and chunking. Exotic formula variants such as BM25L and BM25+ add much less. For Russian, with its cases, this is especially noticeable: if you don't reduce words to their base form, "договора" and "договору" are different words to the index.

What to do in practice

  1. Pick one BM25 library, pin its version, and tune thresholds and hybrid weights within it.
  2. Tune the k1 parameter and other settings separately, otherwise you will be comparing a tuned model with an untuned one.
  3. Improve tokenization and chunking first, and only then think about formula variants.
  4. Plan for index rebuilds, or tolerate statistics drift if updates are frequent.
  5. Recompute hybrid weights after fine-tuning the embedder.

Limitation of the material: this is the view of a single practitioner in one specific domain, legal texts, and the EntityQuestions figures are no substitute for testing on your own collection. But as a checklist of "what breaks when moving from a notebook to production," the breakdown is useful to any team whose RAG loses exact identifiers.

Story

Key Questions Answered

What are the key details on BM25 as a precomputed table?

Imagine that every word in every document has a score assigned in advance: how important that word is for that text. Search then boils down to adding up the ready-made scores of the words in the query. That is why BM25 is so fast. Technically, the weight of a "word–document" pair depends on the word's frequency in the document, the document's length, and global corpus counters, but not on the query. That means it can be computed at indexing time and stored in a sparse matrix.

What are the key details on "BM25" is not a single formula?

There is no single standard. Kamphuis et al. (ECIR 2020) compared eight common variants: Robertson's classic, Lucene, ATIRE, BM25L, BM25+ and others. They differ in how they compute IDF, how they normalize term frequency, and how they handle negative values (which appear when a word occurs in more than half of the documents). Practical consequences: The same documents get different numbers in different libraries, so comparing them across systems is meaningless.

What the benchmarks show and how to blend with a neural model?

The numbers were given above: on EntityQuestions, BM25 found the right answer in 72% of cases and neural search in 49.7%. Neural models handle rare terms poorly (surnames, case numbers, part numbers) and "prefer" short fragments; on long texts they underperform. For a hybrid, the author recommends blending BM25 and the neural model with a weighted sum of scores rather than by ranks (the RRF method, which only considers a document's position in the list).

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…