I Wrote 61 Tests for My AI Mentor. Then I Found Out You Can't Trust the Tests Either

A month ago, a developer integrated an AI mentor into his learning platform. The model sees the course lessons, the student's progress, and their latest attempt at an assignment. To choose a model and a prompt based on numbers rather than gut feeling, he built a test bench of 61 automated tests. A week later, it turned out the bench had bugs of its own. It marked cut-off answers as passing, let finished solutions through under different variable names, and failed correct refusals. This article covers how the bench works, what it helped uncover, where it created false confidence, and why the developer stopped trusting the model entirely.

Tests: 61 questions across 13 groups

The bench consists of a question plus a set of automated checks: substrings, tool calls, response length, language. Each question includes a student message and a set of expectations. The 61 questions are split into 13 groups: lessons (12), prompt attacks (10), epistemic discipline (6), student progress (5), student code (5), engineering reasoning (5), topic boundaries (4), engineering accuracy (3), lesson references (3), response format and language (2), honesty (2), dialogue context (2), tone (2).

The mentor works with three tools: search across 272 real course lessons, student progress (a fixture on the bench), and the student's latest attempt (also a fixture). On the bench, the student has a Reverse function that mutates the source slice in place, even though the contract requires it to return a new one.

Three models: DeepSeek, YandexGPT, Qwen

The developer chose among models available from Russian servers. Claude and OpenAI were out: their APIs reject requests by IP. Yandex AI Studio offers three options: DeepSeek V4 Flash, YandexGPT Pro, and Qwen3.6 35B.

First run on 32 questions: • DeepSeek V4 Flash: 31/32 checks passed, 5 of 5 questions on student code, 6.5-second latency. • YandexGPT Pro: 28/32 checks passed, only 3 of 5 on code, 1.4-second latency. • Qwen3.6 35B: 2/7 checks passed, run aborted.

YandexGPT failed the main thing: it didn't call the tool when asked to analyze the student's code. Qwen didn't fit the configuration — its reasoning mode with a 1024-token limit needs the entire budget just for thinking.

On cost: DeepSeek came to 25.6₽ for 32 questions, YandexGPT to 116₽. The difference isn't in the base price list but in cache pricing: DeepSeek's cached input costs four times less, while YandexGPT charges the same rate as regular input.

Where the bench lied to the developer

On a base of 57 questions, DeepSeek V4 Flash ran three consecutive passes: 50, 48, and 50 checks. The developer sorted the failures into three buckets: false bench errors (fixed), length overruns (an intentional limit), and genuine model errors.

Three main recurring errors: • Handing out a finished solution on a help request — one time in three, the model gave a full solution instead of a hint. For a platform where the assignments themselves are the product, this is the worst possible failure. • A lecture instead of an answer — asked "What does make do in Go?", it answered with a course lecture three times out of three: capacity, two code blocks, a var declaration. • Listing possibilities instead of diagnosing — on a "connection reset by peer" error, it listed typical causes, even though the prompt instructs it not to fill gaps with words like "usually" and "most likely".

Two hidden bugs in the bench itself: 1. A cut-off answer counted as complete. The provider returned finish_reason: length, but the bench didn't check for it and marked it a pass. 2. DeepSeek's default reasoning mode. The developer never passed the parameter, so the bench was running in a hidden reasoning mode. With reasoning_effort: none, the run was 10% cheaper and 40% faster, but the failures were different. With reasoning_effort: high, cost rose 55%, latency rose 65%, and a long tail appeared: on a Kubernetes question, the model made 11 search calls in 36 seconds for 13₽.

The developer left DeepSeek V4 Flash on default settings. The figure 50 out of 57 isn't a percentage of correct answers — it's the number of automated checks that passed. He reads every response by hand, because a substring check can both miss a bad answer and fail a good one. The most useful outcome wasn't picking a model — it was the list of places where the bench itself created false confidence.