A Teacher Built a Bot to Grade the Unified State Exam: The Model Labels, the Code Calculates Scores

What happens if a language model is trusted to assign a student a score for an extended answer? It grades generously: the student “generally understands social studies,” while FIPI's criteria are strict. An example about “country Z” instead of Russia earns zero, and an outline missing three mandatory points loses the entire criterion. A teacher and Unified State Exam expert explained how he spent two years building a bot that accounts for this.

The model labels, the code calculates

The core principle is that the model does not write, “I am awarding 2 points because…” It fills out a JSON schema through tool_use: which examples are specific, which statements are correct, and which outline items are essential. Conventional code then takes over: it recalculates scores according to the criteria, checks everything that can be verified without the model, and launches secondary checks that can only lower the grade.

Each task has its own plugin: a handler, an evaluator, and a specification. There is no universal prompt. The pattern is visible in the amount of code: task 17 takes 942 lines, while tasks 24 and 25 take 6,800–7,200 lines each. The more complex the criteria, the more code is needed around the model—not a longer prompt.

How to avoid punishing an innocent answer

In task 19, the model treated a restatement of a government body's function as an example of that body in action. The author added a second call with a separate schema, but the first version triggered everywhere and deducted points for families and citizens, even though the task did not mention any government body. The secondary check is now activated only by explicit markers in the task text. In addition, an example is rejected only if at least two of four negative flags match. The lesson: protection against inflated scores can itself produce undergrading, and its activation scope matters just as much as the check itself.

In task 24, the scores for the first and second criteria are calculated entirely by code: lemmas are generated with pymorphy2 and compared against 185 reference outlines. Synonymous headings, such as “attributes” and “features,” are normalized to a single form. The test is simple: every reference outline must receive the maximum score. Before synonym handling was fixed, that was true for 147 of 185 outlines; now it is true for all of them.

A point can be deducted for a factual error only after passing four filters: a closed list of error types, an exact quotation of at least 8 characters, two independent runs, and a separate verifier. The last filter is necessary because two identical outputs from the same model do not constitute two independent opinions: on reference outlines, both runs consistently found the same nonexistent error.

An honest failure instead of a fabricated zero

The most insidious bug is a silent zero: the model fails to respond, the code substitutes a zero, and the student sees what looks like a normal assessment. In task 25, there were 16 such zeros across 110 cases, and half matched the reference score, entering the metrics as exact matches. Now every failure sets an ai_unavailable flag: no score is shown, the attempt is not deducted, and the administrator receives an alert. A free attempt is deducted only after the result has been delivered.

The author openly acknowledges the unfinished parts: in tasks 21 and 23, a failure still results in a simplified score, while the backup YandexGPT model must be enabled manually.

Cost and practical takeaway

A November run across 393 answers produced 497 calls and cost $8.87: an average of 1.26 calls, 2.3 cents, and 9 seconds per check. Secondary checks—not long grading calls—are the most expensive component. Prompt caching covers up to 66% of the input, but it does not activate for the short prompts used in tasks 17, 21, and 22.

For anyone building LLM-based assessment: require the model to return a structured response with explicit flags, keep arithmetic in code, make secondary checks one-way, require independent runs, and distinguish a failure from a zero. On newer Claude versions, forced tool_choice returns a 400 error, so pass auto and ask the model to invoke the tool at the end of the prompt.