Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

RAG on Factory Specifications: 84% Accuracy, but 161 Supports Instead of Two

The PSM team loaded 44 documents on the AGBa1000 gas-piston power plant into a RAG system and tested it on 188 questions: accuracy rose from 77.7% to 84%. But the average hides the main point. The parser merged the columns of a scanned table, and instead of two generator supports the assistant saw 161. The model made another error by mistaking a nonexistent fuel tank for a real one.

AI-processed from Habr AI; edited by Hamidun News
RAG on Factory Specifications: 84% Accuracy, but 161 Supports Instead of Two
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Index scanned tables row by row, together with the column header
Key metric #2Link the plant, the engine, and document versions explicitly
Check the reference answersin 19 cases it was the test, not the AI, that was wrong

A typical corporate RAG story: the demo is impressive, ten polished answers in a row, and it seems ready to launch. The PSM team, which is building an assistant for engineers working with documentation for power equipment, decided not to trust the impression and to count errors instead. It turned out that "found the right document" and "answered correctly" are entirely different things.

What Was Tested and How

For the pilot, the team took the AGBa1000 gas-piston power plant with a capacity of 1 MW. The knowledge base included 44 documents: 29 PDFs, 12 Word files, and 3 Excel files. Half of the documents are in Russian and half in English, but by index volume 94% of the chunks are in English: the main manuals are written by component manufacturers. In total this produced 27,007 search chunks.

The control set consisted of 188 questions with reference answers: equipment parameters, operating modes, configuration, and maintenance schedules. The check was automatic: numbers were compared against the reference, and meaning was assessed by an independent model. This is a convergence test, not an expert acceptance review, and the team says so openly.

Two changes raised accuracy from 77.7% to 84%. Table rows began to be indexed as standalone facts, and the models began receiving 10 context chunks instead of 5. The right document now appears in the top ten results in 96.3% of cases (up from 94.1%), and in first place in 71.3% (up from 62.8%). False "no data" refusals fell from 11.7% to 6.9%.

But there is a nuance inside the average: on 26 questions the system started answering correctly, while on 14 it got worse. Moreover, when plant specialists checked some of the reference answers against primary sources, the reference itself was wrong in 19 cases. Before holding the model accountable, check your test data.

Three Errors Invisible in the Percentage

161 supports. The database contained a technical letter from the generator manufacturer with a table: for 800–1120 kW, gas, and COP mode, 2 supports are required. The assistant first replied that there was no data. In another run, the scan parser merged adjacent columns: the power range turned into 16,000–16,100 kW, and the number of supports into 161. The LLM did not hallucinate here; it faithfully retold what the index fed it. The error originated at the parsing stage.

Two unlinked documents. On the question about load steps, the operating manual says only "stepwise." The specific figures (no more than 10% up to half of the power and no more than 5% above) were in the engine manufacturer's letter, which did not mention the AGBa1000 plant. The assistant did not make the connection. The team also debated whether the parameters of a bare engine can be transferred to a complete unit at all: in practice, load acceptance rate depends on gas quality, temperature, and operating mode. A correct answer should cite the engine manufacturer's recommendations with a reference and explicitly state the limits of applicability.

The nonexistent tank. The user was asked to calculate the plant's autonomy on a full fuel tank. The assistant found the gas consumption (161, 229, and 294 m³/h), wrote the formula, and asked for the tank volume. But the plant is powered from a gas main and has no tank. The system accepted a false premise and started calculating. For an assistant that managers and service specialists will use, this is a serious defect.

When Documents Contradict Each Other

A separate finding: official documents contradict each other. The plant's data sheet gives continuous-mode power as 1150 kW, while the engine manufacturer's document gives 1280 kW. The manual prescribes changing spark plugs every 1,500 engine hours, while the engine service instruction says every 5,000.

You cannot pick the newer file, take the most common value, or average. The only sensible option is to show the conflict with section and page references and ask an engineer for confirmation. This turns RAG into a tool for preliminary documentation audit, although people still have to resolve the discrepancies.

What to Take Away

First: scanned tables should not be handed to a general text parser. Index them row by row while keeping column headers, or validate them with specialized models. Row-by-row indexing raised the share of answers with correct numbers from 70.8% to 78.3%, but complex scans with drawings and handwritten notes still fail.

Second: a folder of files is not a knowledge base. You need explicit links between the product, components, configurations, and document versions; otherwise a found fact remains useless.

Third: teach the assistant to spot false premises and source conflicts. Sometimes the best answer is to explain why the calculation cannot be done this way.

And the main limitation: the 84% figure comes from automatic testing of a single plant, and some reference answers turned out to be wrong. It is a good working benchmark, but not proof of production readiness. For engineering documentation, measure not average accuracy but error types: they show where the problem lies, whether in parsing, links between documents, or answer logic.

Story

Key Questions Answered

What Was Tested and How?

For the pilot, the team took the AGBa1000 gas-piston power plant with a capacity of 1 MW. The knowledge base included 44 documents: 29 PDFs, 12 Word files, and 3 Excel files. Half of the documents are in Russian and half in English, but by index volume 94% of the chunks are in English: the main manuals are written by component manufacturers. In total this produced 27,007 search chunks.

What are the key details on Three Errors Invisible in the Percentage?

161 supports. The database contained a technical letter from the generator manufacturer with a table: for 800–1120 kW, gas, and COP mode, 2 supports are required. The assistant first replied that there was no data. In another run, the scan parser merged adjacent columns: the power range turned into 16,000–16,100 kW, and the number of supports into 161. The LLM did not hallucinate here; it faithfully retold what the index fed it. The error originated at the parsing stage. Two unlinked documents.

When Documents Contradict Each Other?

A separate finding: official documents contradict each other. The plant's data sheet gives continuous-mode power as 1150 kW, while the engine manufacturer's document gives 1280 kW. The manual prescribes changing spark plugs every 1,500 engine hours, while the engine service instruction says every 5,000. You cannot pick the newer file, take the most common value, or average. The only sensible option is to show the conflict with section and page references and ask an engineer for confirmation.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…