AI Mentor Author Audits His 61 Tests and Finds the Test Harness Was Lying

A month ago, a learning-platform developer built an AI mentor into the platform and assembled a harness of 61 automated tests for it, so that decisions would rest on numbers rather than gut feeling. A week later it turned out that the harness itself was wrong: it counted truncated answers as passes, let ready-made solutions through under different variable names, and failed correct refusals. A week after that, the model worked flawlessly on the harness, but not in production on the same class of questions. The author described the story on Habr.

What was tested and why

The mentor on the platform is not an abstract chatbot. It sees the lessons, the student's progress and his latest attempt at the assignment. That sets the quality criterion: a tutor is judged not by whether he knows the answer, but by where he stops. Does he point to the line with the error, or rewrite the task for the student? Does he say "I don't know, let's check," or confidently make something up?

The harness was supposed to measure exactly this behavior. It was meant to help choose a model and tune the prompt, so there would be no guessing which version is better. As long as the numbers can be trusted, the setup works. The problems began when the checks themselves were checked.

Where the harness lied

The author names three errors, and each one cuts in its own direction. The announcement does not disclose implementation details, so what follows is our reading.

• A truncated answer counted as a pass. The check apparently looked for the required features in the text but did not require the answer to be complete. A model cut off by the token limit got a pass.
• A ready-made solution under a different variable name got through. The ban on handing out solutions most likely relied on text comparison. Rename a variable and the match disappears, while the student gets the full code.
• A correct refusal failed. The check expected a particular form of refusal and did not recognize a correct answer worded differently.

The first and second errors produce false passes; the third produces a false failure. That is why the overall pass rate says almost nothing: the errors partly offset each other, and the final number looks plausible.

The harness is clean, production is not

The most unpleasant part came later. After the checks were fixed, the model behaved flawlessly on the harness, but in production it did not on the same class of questions. This is the classic gap between a synthetic test set and real traffic. Live students write messier, submit unfinished code, pressure the mentor, and phrase requests differently from the test's author. A set of 61 cases does not cover that variety.

The author's conclusion is harsh: he stopped trusting the model altogether. The headline implies that the tests cannot be trusted either. In practice, this means changing the architecture of trust: the mentor's behavior has to be constrained and verified at the system level, rather than hoping that a good prompt and a green harness guarantee the result.

What you can take away

• Check the evaluator. Run the harness on answers that are known to be bad: truncated ones, ones containing a ready-made solution, ones with renamed variables. If the harness lets them through, the numbers cannot be trusted.
• Compare by meaning. For code, that means identifier normalization, structural parsing or running the solution, not substring search.
• Count false passes and false failures separately. A single pass rate hides errors of both signs.
• Feed production back into the set. Every real failure should become a new test.
• Do not rely on the model in critical places. It is more reliable to enforce a ban on handing out full solutions as an output check than as a request in the prompt.

This story is useful to anyone building agents and LLM assistants with quality metrics. The evaluator is a software component just like the model itself, and it makes mistakes no less often. The author publishes the full breakdown with examples in the original article; here we have retold only what the announcement states.