Qdrant and Hybrid RAG: Searching Corporate Documents Without the Cloud or Data Leaks

Corporate document search is hampered not by a lack of data, but by the fact that the data is scattered across PDFs, scans, spreadsheets and email threads. Hybrid RAG proposes an architecture that lets companies search internal archives locally, without sending sensitive information to the cloud.

Where search breaks down

Standard RAG works well on a tidy, uniform knowledge base: an FAQ, a single policy document or a set of clean text files. In real business, though, documents come in different formats and two languages, and often contain not just text but also tables, codes, part numbers and scanned pages. In that environment, semantic search alone is not enough: a query about claims against a counterparty, or about code 8471300000, requires understanding the meaning of the question while also precisely locating specific numbers and wording.

This is where Hybrid RAG differs from the classic approach. The system does not simply "feed" the archive to a language model. It first assembles relevant context using several types of search. This matters most for logistics, banks, customs and law firms, where users need not a general summary but a precise answer tied to a specific document, clause or page. That is why the key issue is not only retrieval but also how the retrieved context is verified before the model answers.

How the stack is built

In the described setup, documents first go through structural parsing. Docling prepares scans and layout, while the multimodal model Qwen2.5-VL helps read complex tables, handwritten notes and poorly recognized pages. Then the embedding model BAAI/bge-m3 turns documents and the user's question into two representations at once: dense for semantic search and sparse for exact matches. Qdrant stores both types of vectors and merges the results via RRF, so nobody has to balance weights between the two strategies by hand.

• Docling normalizes incoming files and prepares the document structure
• Qwen2.5-VL helps parse scans, tables and complex visual elements
• BAAI/bge-m3 builds dense and sparse vectors for documents and queries
• Qdrant runs the hybrid search and merges the results via RRF
• A rerank layer filters out irrelevant fragments before the model answers

Next comes a two-stage rerank: a fast filter followed by a more precise cross-encoder check. What reaches answer generation is not the whole archive but a small set of fragments that actually answer the question. This pipeline reduces the risk of hallucinations and lets the system avoid making things up when the needed fact is simply not in the database. That is the difference between document-grounded search and a polished but unreliable summary.

Why self-hosted matters

The article's main thesis is that the problem cannot be solved by simply plugging in a cloud LLM. In regulated industries, data must not leave the company perimeter during indexing, search or answer generation. The author therefore bets on a self-hosted stack: a local Qdrant, a local Langfuse for tracing, LangGraph for explicit state management and Haystack for quality checks before production. Otherwise, the search system itself becomes a new compliance and audit risk.

"Not enough data" is better than a confident hallucination.

One practical takeaway concerns infrastructure. Qwen2.5-72B-Instruct delivers the best quality, but it needs two A100 80GB GPUs. A more realistic starting option for business is Qwen2.5-32B-Instruct on a single L40S: by the author's estimate, it delivers about 90% of the larger model's quality at a noticeably lower cost. That makes Hybrid RAG not a lab toy but a sound architecture for companies that need internal search with a clear total cost of ownership.

What this means

Hybrid RAG is becoming a practical way to bring corporate archives to life without compromising on security. For business, it is a path from chaotic searching through email and SharePoint to a system that finds the right document in seconds, shows the source of the answer and does not send internal data to an external API.