Qdrant and Hybrid RAG: corporate document search without the cloud or leaks

Corporate document search is hindered not by a lack of data, but by the fact that this data is scattered across PDFs, scans, spreadsheets and correspondence. Hybrid RAG offers an architecture that allows searching through internal archives locally, without exposing sensitive information to the cloud.

Where Search Breaks Down

Standard RAG works well on clean, homogeneous databases: for example, FAQs, a single regulation, or a set of clean text documents. But in real business, documents live in different formats, in two languages, and often contain not just text but tables, codes, article numbers, and scanned pages. In such an environment, semantic search alone is not enough: a query about claims against a counterparty or code 8471300000 requires both understanding the meaning of the question and finding specific numbers and formulations precisely.

This is where Hybrid RAG differs from the classical approach. The system does not simply try to "feed" the archive to a language model, but first collects relevant context from several types of search. This is especially important for logistics, banking, customs, and law firms, where the user needs not a general paraphrase, but an exact answer backed by a specific document, section, or page.

This is why what matters is not just the search itself, but the ability to verify the context found before the model responds.

How the Stack Works

In the described scheme, documents first undergo structural parsing. Docling prepares scans and layout, while the multimodal Qwen2.5-VL model helps read complex tables, handwritten notes, and poorly recognized pages. Then the embedding model BAAI/bge-m3 transforms documents and the user's question into two representations simultaneously: dense for semantic search and sparse for exact matches. Qdrant stores both types of vectors and combines results through RRF, so there's no need to manually balance weights between the two strategies.

• Docling normalizes incoming files and prepares document structure
• Qwen2.5-VL helps parse scans, tables, and complex visual elements
• BAAI/bge-m3 builds dense and sparse vectors for documents and queries
• Qdrant performs hybrid search and merges results through RRF
• A reranking layer filters out irrelevant fragments before the model responds

After this, reranking is applied in two stages: quick filtering and more accurate cross-encoder verification. The answer generation receives not the entire archive, but a small set of fragments that actually answer the question. This pipeline reduces the risk of hallucinations and allows the system not to speculate if the base simply doesn't contain the needed fact. This is the difference between document-backed search and beautiful but unreliable paraphrasing.

Why Self-Hosted Matters

The main point of the article is that the problem cannot be solved by simply connecting a cloud-based LLM. In regulated industries, data must not leave the company's perimeter during indexing, search, or answer generation. Therefore, the author bet on a self-hosted stack: local Qdrant, local Langfuse for tracing, LangGraph for explicit state management, and Haystack for quality verification before production. Otherwise, the search system itself becomes a new compliance and audit risk point.

"Not enough data" is better than confident hallucination.

A separate practical conclusion concerns infrastructure. Maximum quality comes from Qwen2.5-72B-Instruct, but it requires two A100 80GB GPUs. A more realistic starting option for business is Qwen2.5-32B-Instruct on a single L40S: according to the author's estimate, it delivers around 90% of the quality of the older model, but costs significantly less. This makes Hybrid RAG not a laboratory toy, but a clear architecture for companies that need internal search with understandable cost of ownership.

What This Means

Hybrid RAG becomes a practical way to bring corporate archives to life without compromising on security. For business, this is a path from chaotic email and SharePoint search to a system that in seconds finds the needed document, shows the source of the answer, and does not send internal data to an external API.