Developer achieves 73–86% agreement with an expert on engineering questionnaires

An engineer needs to complete a questionnaire for an industrial instrument supplier, specifying pressure, materials, climatic suitability and explosion protection. The answers are scattered across hundreds of pages of technical specifications. It sounds like a ready-made job for an LLM: find the relevant passage and copy the value into a table. In the project described on Habr, the first version filled in only about a third of cells correctly.

Over two months, the author brought the system to 73–86% agreement with an engineer across three projects, counting only rows the engineer had answered. But the analysis of errors is more useful here than the headline figure. The system confused closely related parameters, tried to guess the department's internal decisions and treated rows that required no engineering response as questions.

Authentic quote, wrong answer

The first version followed a familiar approach: for each cell, retrieval returned twelve passages from the technical specification, and the model produced a value and a quote. The code checked whether the quote actually existed. That proved insufficient.

A simple example: the system writes “Warranty: 30 years.” The number really is in the document, but it refers to the unit's service life. The actual warranty—36 months—is nearby. The source check passes, and the engineer gets an incorrect answer.

The author handed questionnaire completion over to rules and a reference database, leaving the model to extract facts. Where the rules applied, 86–92% of answers were correct. Across all cells in two other projects, the result was 58%. The rules, too, carried project-specific assumptions from one project to another.

Better retrieval alone did not guarantee an improvement. After a small model was fine-tuned, the relevant clause ranked first in 63% of rows, up from 52%, but the number of correct values did not increase. The next stage still grabbed the first passage from which it could extract anything. Additional conditions helped: checking a key word in the parameter and the unit used for the duration.

For teams building retrieval over corporate documents, this is an important lesson: measure the entire pipeline through to the final answer. Better search results are useless if the next step handles them incorrectly.

Some knowledge lives within the department

An engineer's review revealed a deeper problem. Among nonstandard rows in the first four projects, the author could not find answers in the documents in 65–83% of cases. A manual review of twenty such rows found that about half reflected departmental decisions that had been carried from one questionnaire to the next for years.

The model could not retrieve that information from the technical specification. It was not there. Past questionnaires and departmental standards were therefore added to the system. Matching answers from two projects provided a stronger basis for suggesting a similar answer than a single example did. Parameters tied to a specific operating environment still require caution: a value from another project may be unsuitable.

In the working interface, each answer comes with a link to the document clause, a quote and a page with highlighting. If there is no supporting evidence, the row stays empty and is added to the questions for the customer. The system presents conclusions drawn from standards separately, with explanations, for an engineer to review.

A product like this helps gather and verify information. Its practical value also depends on how easily a specialist can confirm a value, correct it and save the decision for the next project.

Impressive percentages need scrutiny

The author found several errors in the evaluation itself. Checkboxes on forms were read incorrectly. Answers from the project under evaluation leaked into the database through old standards, making the result look like 71% instead of the honest 61%. Fuzzy matching accepted IP54 as IP65, even though these are different protection ratings.

Finally, more than half the form's rows turned out to be requirements, lists and signature fields. Excluding them raised completion from 45 to 81%. Not a single correct answer was added: the denominator was corrected.

The final 73–86% also comes with a limitation. Although the system had not seen the answers for the three evaluation projects, the author tuned the rules using their errors. For two projects, tables from scans were manually transferred into Excel. This is a test-environment result; evaluation on a new project is still ahead.

The practical takeaway for developers: before switching LLMs, check the form structure, knowledge sources, retrieval and evaluation method. For a product manager, the next question is how much time the engineer spends reviewing and correcting answers. That is what will show how much the system helps in practice.