VerifierBench: 19 LLMs Wrote Verifiers, and the Best Scored Only 62.1

Agents need a way to know they have solved a task correctly. Without one, they cannot finish a job on their own, and they cannot filter synthetic training data. The VerifierBench authors tested how well LLMs handle this themselves: given an instruction and an available environment, the agent writes an executable verifier that must accept every correct solution and reject every incorrect one. The benchmark includes 222 tasks from six agentic benchmarks and 4993 hidden test cases. Across 19 models, normalized balanced accuracy ranged from 10.6 to 62.1.

What exactly VerifierBench tests

The task is set up strictly. The verifier gets the same information as the solving agent: the task text and the environment. From that, it must build a program that issues a verdict on someone else's solution. The tasks cover three types of checks: tool calls (tool-call), the final state of the environment (state-based), and code.

The hidden tests are built to catch more than obvious misses. Each task has about 22 cases on average. They include alternative valid solutions, meaning other ways of reaching the correct result, and attacks that try to fool the check. The authors run the original verifiers and test sets through Verifier4Verifier (V4V), an audit consisting of six checks. This keeps the reference from becoming a source of its own errors.

Key parameters of the work:

• 222 tasks from six agentic benchmarks
• 4993 hidden test cases, including alternative solutions and attacks
• 19 models in a single agentic harness
• normalized balanced accuracy: from 10.6 to 62.1
• the project page and dataset are public

A verifier that lets no one through

The most interesting finding is the imbalance. Generated verifiers consistently reject incorrect solutions better than they accept correct ones. At first glance this looks like a strength: a strict check seems safe. The authors show that the real picture is different. Most false rejections in the analyzed records come from verifiers that accept none of the tested solutions to a task. Such a verifier formally does half the job well and fails entirely at the other half.

For pipelines, this is a serious problem. If a verifier filters synthetic data, excessive strictness silently discards correct trajectories. If it serves as the success signal for an agent at runtime, the agent may keep redoing a result that is already correct.

A review of individual cases shows where the strictness comes from. Verifiers bake in unverified assumptions about the format of the solution, about which tools must be called, and about what the code should look like after the edits. Any solution structured differently from what the check's author imagined gets rejected, even if it solves the task.

Why self-testing does not catch the error

One might expect the agent to notice the flaw itself by running its verifier on trial solutions. In practice, the errors survive self-testing, and the authors name two reasons. First, the agent tests solutions that share the same mistaken assumptions as the verifier itself. It writes a check for one answer template, then generates an answer from the same template, and everything lines up. Second, some agents submit a verifier without confirming that even one correct solution passes through it.

How to test the checkers themselves

The authors' conclusion is practical: it is not enough to confirm that a verifier accepts one correct solution. You also need to check whether it accepts the other solutions the task allows. For teams building agentic pipelines or assembling synthetic datasets, several steps follow from this.

First, every generated verifier needs at least one known-correct example that passes the check, or it may turn out to be a "blank wall." Second, the test set should contain alternative correct solutions with a different format, a different order of calls, and a different way of modifying the code. Third, it is worth tracking the false rejection rate separately rather than relying on a single overall accuracy figure: a skew toward strictness is easy to hide in an aggregate metric. Finally, the reference itself must be audited, and the V4V approach offers a ready-made scheme for that.

The spread from 10.6 to 62.1 also shows that verifier generation is still far from a solved problem. Even the best result leaves a notable margin of error. It is therefore too early to trust agent-based evaluation without independent oversight, while the public dataset gives teams a convenient way to compare models on this specific skill.