AI Reviewer Approved an Agent's Mistake: One Success Rate Hid Three Different Defects

Picture a typical scene from an online store's support desk. A customer writes to an agent: "Cancel my previous order." But the customer has two orders. The correct behavior is obvious to any human operator: ask which one exactly. The agent, however, silently picked one, canceled it, and cheerfully reported that everything was done. And a second AI, which was checking the responses, gave it a high score.

This story from a write-up on Habr is notable because it contains not a single outright lie. And yet the outcome is bad.

Why the Reviewer Missed the Problem

The reviewer compared the agent's message with the state of the database. The order really was canceled, the message matches that, so everything is honest. Formally, it is right: the agent did not lie about what it did.

The problem is what the judge was actually asked to evaluate. Its task was to check how accurately the agent described its own action. The question "did the agent have the right to act without clarification" was not part of that task. So the positive score let through a defect that a separate programmatic check caught.

This is a classic trap: the judge answers the question it was asked, not the one you had in mind.

Three Different Defects Under One Percentage

The author started with a different hypothesis: he thought a short addition to the instructions would degrade the agent's performance. The experiment did not confirm this. However, reviewing the logs showed that behind the overall success figure hide three distinct problems, and they are fixed in different ways.

First: the agent performed the action without the required clarification, and the reviewer, which looked at explanations, did not stop it. This calls for checking the behavior itself, not the text of the response.

Second: the test required choosing a response code according to a rule that the instructions explained incompletely. This is an error not of the model but of the setup: the agent was penalized for something it was never told. It is fixed by correcting the instructions or the test.

Third: the agent performed the operation correctly, but the reviewer invented a reason for rejection. That is, the judge made up a complaint. Here you need to fix the evaluation criteria and constrain the judge so it does not assess things that are not in the rules.

If you look only at the average percentage, all three cases look the same: "test failed" or "test passed." Choosing the right remedy from such a number is impossible.

What to Do in Practice

First, separate the evaluation of the action from the evaluation of the explanation. The fact of the order cancellation, the call of the right tool, the presence of a clarifying question are better verified by code against logs and system state. This is deterministic and does not depend on the model's mood.

Second, for ambiguous commands, make clarification a mandatory step in the workflow. If the user has two orders and the request has no identifier, the system should require a question. Relying only on wording in the prompt is risky.

Third, give the AI judge a narrow and explicit rubric. The less freedom it has to "fill in the gaps," the less often it invents reasons for rejection or, conversely, approves what it should not.

Fourth, sort failures by type. For each failed run, answer three questions. Did the agent make a mistake? Was the instruction or test incomplete? Did the judge make a mistake? Only after that does it make sense to change anything.

Limitations

This is an analysis of one set of experiments with one type of scenario, so no universal numbers can be derived from it. But the scheme itself is widely applicable: wherever one AI checks another, it is useful to ask what exactly is being checked and what remains off-screen. A single overall success rate is convenient for a report, but for debugging agents it is almost useless.