RAG Over 11,000 Files Confused Part Numbers and Returned Data From the Wrong Machine

A sales-team bot that answers from technical documentation makes costlier mistakes than an online store's support chat. A manager gets a figure, passes it to a client, and the client orders the wrong machine. That is exactly the case described on Habr: 11,000 technical documentation files, more than 100 sales managers, and a RAG bot that confidently quoted figures from the datasheet of a neighboring model.

What we know about the case

The Habr post is currently available only as an announcement in the feed, so we rely solely on what it states. Everything else in this text is general engineering logic for such systems, not a retelling of the author's measurements.

• Corpus: about 11,000 technical documentation files.
• Audience: more than 100 sales managers.
• Symptom: the bot answers confidently but takes parameters from the datasheet of a different machine.
• Problem class: queries by part number and exact model designation.
• Stated outcome: the author describes what eventually worked, along with an analysis of the causes.

Why vectors confuse part numbers

Dense embeddings compress a text fragment into a vector that reflects its meaning. For a question like "what is the spindle power of the machining center," this works well. For a query by part number such as "XYZ-4200-B," the picture is different. The tokenizer splits such a string into subwords, and the embedding model has no obligation to treat the difference between "4200-B" and "4200-D" as significant. In vector space, such designations end up as neighbors.

Product datasheets from the same series make the problem worse. They are written from one template: the same sections, the same wording, the same tables. Only the numbers differ. To an embedding, this is almost the same document, and search happily returns chunks from several models in the top 5.

Then ordinary LLM mechanics take over. Given a plausible fragment in its context, the model quotes it. It does not know the datasheet is the wrong one: the query contains a part number, the context does not, and nothing checks for that mismatch. The error arises at the retrieval stage, not at generation, so editing the system prompt is almost useless here.

What to check in your own RAG

The toolkit for such tasks is well known, and each technique is worth testing against real manager queries.

First: hybrid search. A lexical index such as BM25 runs alongside the vector one, and the results are merged via Reciprocal Rank Fusion or weighting. An exact match on the part number string gets a weight that a vector cannot give it.

Second: extracting identifiers from the query. If the question contains a part number, it can be recognized with a regular expression or a model reference list and used as a hard filter on document metadata. Then the wrong datasheet simply never makes it into the results.

Third: context inside the chunk. If, during chunking, you add the model name, series, and file name to each fragment, a table of figures stops being anonymous.

Fourth: a reranker. A cross-encoder over the top-N candidates matches a query to a fragment better than comparing two independent vectors.

Fifth, and cheapest: a check before answering. If the part number from the query is not found in the sources that went into the context, the bot should say "not found" rather than guess.

For testing, assemble a set of real questions with part numbers, including pairs of similar models, and measure the share of cases where the right document lands in the top-k. This metric will reveal the problem before a client notices it.