LLM Service Classifier Rose From 73% to 94%, but a Three-Node Chain Still Delivers Just 75%

Any team that has run an LLM in production knows this disappointment. On the test set the model delivers 98%, but on real data it comes out at 73%. The author of the analysis on Habr believes that it is rare to get whole-system quality above 90% in production, so that is the number to aim for. Some projects, because of the high cost of error, simply won't add up.

The task: 50 classes and one reference book

The example comes from real work. Given a service name, you need to select a code for form 402. The classification is flat, with 50 classes, and each type of service corresponds to exactly one code. For example, "842 — Software development and installation" or "7123 — Freight transportation by road." The task looks simple, but it shows how a single node behaves.

The first instinct is to keep adding exceptions to the prompt. This bloats the prompt and creates three problems. First, the model may "lose" part of a long context: the information is in the input but not in the answer. Second, if one prompt contains dozens of different tasks (say, extracting 50 attributes, with a whole paragraph of rules for the amount alone), some of the rules drop out. Third, an analogue of overfitting appears: you tune the prompt to the same benchmark, and it memorizes the examples instead of the general rule. The team stopped at 8 examples and got 81%.

What really improved the node's quality

The author rejects the naive approach of splitting the reference book in half. That produces two candidates, and a third step is needed to pick one code. This is a new place for errors.

What helped was analyzing the logs and talking to the business. It turned out that one document can contain several services. The previous node wrote them as a single line, and the LLM picked one code while the other services were lost. The fix: extract the services as a list, feed each one separately with the necessary document context, and assemble the result with a code for each. Quality rose to 89%. The problem was in the task formulation, not in the model.

The next step is dynamic examples. From the labeled benchmark, a table of "service name vector — class" is built. For a new service, similar ones are found and inserted into the prompt together with the correct codes. This produced 94%. Reaching 100% isn't possible: the table is incomplete, and vectors sometimes match falsely, so unsuitable examples end up in the prompt.

Chains: quality multiplies

The main idea of the second half of the article: for a chain of calls, the final quality equals the product of the quality of the steps. An example from a contact center. Question rewriting works at 95%, RAG search over the knowledge base at 80%, and the final answer at 99%. Total: 0.95 × 0.80 × 0.99 = 75%. The calculation is only valid on one fixed dataset: you can't measure each link on its own set and then multiply.

The temptation is understandable: add orchestration with a stop list of problematic questions. It does cut off some cases, and RAG works better on the rest, say at 89%. But the orchestrator itself also makes mistakes, say in 5% of cases. Result: 95% × 95% × 89% × 99% ≈ 79.5%. There is a gain, but a small one, and the system has become more complex. It did not become perfect: the chain gained one more step with its own error.

What to do in practice

• Measure quality on realistic data, not only on the internal benchmark. A gap of 98% vs. 73% is a typical scenario.
• Before making the prompt more complex, look at the logs and talk to the business. Some "model errors" will turn out to be errors in the task formulation or in the previous step.
• Split the task by meaning (one service — one call), not mechanically.
• Calculate the chain in advance: three steps at 90% give about 73%.
• If the cost of error is high, build a human into the loop. Don't expect 100% from an LLM system.

Limitations

The figures come from a single case by the author, in which he changed the inputs while preserving the overall meaning. There is no independent verification, and the size of the gain from each technique depends on the data. But the logic is universal: system quality is determined by the weakest link and the number of steps, and it is better to accept the 90% benchmark right away.