BM25 vs. RAG: why keyword search and semantic search return different answers

BM25 remains the foundational search mechanism in Elasticsearch and Lucene, but it has a hard limitation: it understands words, not meaning. Against this backdrop, RAG with vector embeddings solves a different problem—it finds relevant fragments even when the query and document have no exact match in phrasing.

How BM25 Works

BM25 ranks documents by three main signals: how often terms from the query appear in the text, how rare those terms are across the entire collection, and whether the document is too long compared to the average. These factors combine into a final score that the system uses to sort results. This approach has held the foundation of classical search for decades because it's fast, understandable, and doesn't require a separate model to interpret query meaning.

The term frequency saturation mechanism is particularly important. If a word appears five times, it usually noticeably increases relevance; if fifty times, the gain barely changes the picture. The k1 parameter controls how quickly this saturation kicks in, and the b parameter controls how heavily to penalize long documents. The IDF layer (inverse document frequency) amplifies rare words and weakens common ones. But BM25's fundamental drawback doesn't disappear with any tuning: the algorithm sees text as a set of tokens and doesn't distinguish context, word order, or meaning.

Where RAG Wins

Vector search, which typically lies inside a RAG pipeline, works differently. The system transforms both documents and the query itself into dense numerical vectors through an embedding model, then compares them by cosine similarity. In the example from the article, OpenAI's text-embedding-3-small model with dimensionality 1536 is used. This allows a query about "finding similar content without exact word matches" to return relevant text even if the necessary words don't appear in the document at all.

"Neither approach is better at everything: they fail in opposite directions."

This is where a practical compromise emerges. BM25 can be set up locally: tokenization, index, arithmetic—and search is ready. Vector retrieval requires API calls at indexing and query stages, plus storing the embeddings themselves. For a small dataset this is trivial, but with hundreds of thousands or millions of chunks, this setup becomes an infrastructure and financial decision. However, it better handles synonyms, paraphrases, and queries where the user expresses meaning rather than exact keywords.

Practice and Trade-offs

The article builds its comparison on a simple Python demo with 12 text chunks covering BM25, TF-IDF, RAG, transformers, Django, PostgreSQL, and other topics. For BM25, the rank_bm25 library is used, and for embeddings, OpenAI's API and standard cosine similarity calculation. The same query is then run through both retrievers to see which fragments land in the top results. This clearly shows: systems answer one question but arrive at results through completely different signals.

• BM25 searches for exact words from the query and easily explains why a document ranked higher.
• Vector search searches for meaning and better catches synonyms and paraphrases.
• BM25 doesn't require GPU, models, or external calls.
• Embeddings require a separate index, model calls, and space to store vectors.
• Hybrid search combines both approaches and has become the standard for production.

The article's conclusion is quite grounded: arguing what's "better" is pointless outside the context of the task. If you need fast, cheap, and transparent search by explicit keywords, BM25 remains very strong. If semantic matching and resilience to different phrasings matter more, dense retrieval wins. That's why in real RAG systems today, both outputs are increasingly combined first, then candidates are passed to the LLM for an answer.

What This Means

For teams building search across a knowledge base, FAQ, PDFs, or internal wikis, this is good guidance without unnecessary magic. BM25 hasn't become obsolete, and RAG doesn't replace classical search. On the contrary, the most reliable systems today are built from both layers: one provides keyword precision, the other provides semantic understanding and paraphrase resilience.