Sber's AutoJudge Identifies the Faulty Agent in 67% of Cases, Up from 55%
In a chain of AI agents, one early mistake, such as a misread number, can carry through to the final answer as if it were settled. Sber AI Lab's AutoJudge assembles its own judges for each run and, from the log, names the agent that went wrong first. On the Who&When benchmark, with the same GPT-4o model, it finds the culprit in 67.24% of cases instead of 55.17%. The whole execution path needs checking, not just the final answer.
AI-processed from Habr AI; edited by Hamidun News
Picture a system of several AI agents. The first one pulled the wrong number from a document, the second honestly calculated a result from it, and the third neatly assembled a report. Each step looks reasonable, but the overall answer is wrong, and the final text alone doesn't reveal who made the first mistake. The applied research team at Sber AI Lab has proposed AutoJudge: a system that assembles a set of "judges" that go through the agents' logs and point to the step and the agent where the error occurred. On Who&When, a benchmark that requires naming the faulty agent from the run history, accuracy rose from 55.17% to 67.24% with the same GPT-4o model.
Why checking the final answer is not enough
While an LLM just answers in a chat, its errors can still be spotted by eye. But once an answer is written to memory, becomes input for the next step, or triggers an action, manual review stops working. That is why hallucination detection, meaning the automatic search for claims that aren't supported by the system's data, is becoming part of the production pipeline.
In RAG, the typical case is called contextual hallucination. A document says: "The study involved 120 people." The model replies: "...evenly split between a control and an experimental group." The number comes from the source, but the even split was invented by the model. What matters is not whether the claim happens to be true, but whether the answer is supported by what the system actually received.
There are three families of methods for checking an individual answer:
- Uncertainty estimation. These look at token probabilities or at how much repeated answers diverge. No separate model is needed, but confidence is not correctness: an LLM can be confidently wrong.
- Probing internal states. A small classifier is trained on top of the model's hidden representations. If the generator is available only through an API, an open surrogate model is used instead. At least 300-500 labeled examples are needed.
- Judge models. These are either a compact BERT-based classifier or a generative LLM acting as a judge. An LLM is more flexible and suits cases where labeled data is scarce, but it costs more. Fine-tuning a judge requires at least 1,000 examples.
To avoid building separate infrastructure for each method, Sber AI Lab created the SIRIN library. It provides a single interface for all three approaches, works with both open and closed models, and looks for problems either in the whole answer or in a specific fragment. It also checks whether the retrieved documents are sufficient to answer before generation: if they aren't, the system can repeat the search or decline to answer.
How AutoJudge checks the agents' execution history
In a multi-agent system, the object of verification is different: it's the entire history of messages between agents, tool calls, and their results. On top of hallucinations, there are errors in planning, tool use, communication, and aggregation. Existing methods for evaluating such histories are usually tied to a specific list of error types and a predefined verification pipeline. When the system or the criteria change, everything has to be redesigned.
AutoJudge takes as input the execution history, an evaluation taxonomy (that is, the list of error types to look for), and a schema for the required output. A meta-agent then analyzes the task and itself creates a set of specialized intermediate judges along with a final aggregator. The judges examine different stages of the work, and the aggregator reconciles their conclusions with the source data. This is not simple voting: if the assessments diverge, the aggregator returns to the original history and resolves the contradiction.
There are two modes. In the main mode, a separate pipeline is built for each run: this costs more, but it adapts better to how the agents behave. In the economical mode, one pipeline is reused across the entire dataset. In the experiments, adapting to each specific run usually proved more useful, especially when the agents behave differently.
Log size is a separate problem. Agent messages together with tool results may not fit into each judge's context. So AutoJudge can work with either the full history or a condensed one: the judges see compressed steps, and the original details are pulled selectively from external storage.
What to do with this in practice
The gain from 55.17% to 67.24% was achieved with the same GPT-4o, so it cannot be explained by a stronger model. The method was also tested on several other datasets: dynamically built pipelines remained competitive and often outperformed fixed configurations.
The authors do not present AutoJudge as a universal benchmark: on narrowly specialized metrics, hand-tuned judges are sometimes stronger. The adaptive mode is more expensive, and compressing the history can lose an important detail.
The takeaway for those building agents: log steps and tool results so they can be analyzed after a run. In RAG, add a context sufficiency check before generation. In chains of several agents, evaluate the whole path, not just the final answer. Then the question "is the answer supported?" becomes a more useful one: "at which step did the system go wrong?"
- Nov 21, 2026VibeCBS collects project nicknames in advance: the agent finds 2.8 times more documents
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026Sakana AI Opens Applied Research Lab for Market and Economic Modeling
- Nov 21, 2026Meta Is Reportedly Preparing the $300 Muse Charm AI Tamagotchi With a Year of Verizon 5G
Key Questions Answered
Why checking the final answer is not enough?
While an LLM just answers in a chat, its errors can still be spotted by eye. But once an answer is written to memory, becomes input for the next step, or triggers an action, manual review stops working. That is why hallucination detection, meaning the automatic search for claims that aren't supported by the system's data, is becoming part of the production pipeline. In RAG, the typical case is called contextual hallucination.
How AutoJudge checks the agents' execution history?
In a multi-agent system, the object of verification is different: it's the entire history of messages between agents, tool calls, and their results. On top of hallucinations, there are errors in planning, tool use, communication, and aggregation. Existing methods for evaluating such histories are usually tied to a specific list of error types and a predefined verification pipeline. When the system or the criteria change, everything has to be redesigned.
What to do with this in practice?
The gain from 55.17% to 67.24% was achieved with the same GPT-4o, so it cannot be explained by a stronger model. The method was also tested on several other datasets: dynamically built pipelines remained competitive and often outperformed fixed configurations. The authors do not present AutoJudge as a universal benchmark: on narrowly specialized metrics, hand-tuned judges are sometimes stronger. The adaptive mode is more expensive, and compressing the history can lose an important detail.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.