AgentProofBench: CMU researchers test whether web-agent verifiers can be reused

Researchers at Carnegie Mellon University, led by Jingjie Ning, have introduced AgentProofBench, a benchmark for web-agent verifiers. It addresses a practical question: can you write an acceptance rule for a result once, then reuse it when requirements change and in real execution? The benchmark covers 97 tasks, 79 task families and four web services. The verifier programs were generated by DeepSeek Flash and DeepSeek Pro.

The idea behind the benchmark is that judging a verifier by verdict accuracy alone is not enough. An agent works in a chain: it receives an instruction, acts on a website, passes a check, and only then is the result delivered to the user. AgentProofBench therefore links paired tests of a "frozen" acceptance rule to agent runs from the same saved initial states. Every test error can be traced to its consequences in execution.

Reuse or rebuild

The first dimension is robustness to changing requirements. On 48 task families, the authors applied two tests: complete-panel and decision-replay. They show whether the original verifier program remains usable when a requirement changes, or whether it has to be rewritten.

• Flash: 82 of 96 original programs survived (about 85%).
• Pro: 79 of 96 survived (about 82%).
• The difference between reuse and rebuild on the BA metric for Flash lies in the interval from −5.47 to 1.57 percentage points.
• For Pro, the same interval is from −10.16 to 0.00 pp.

The Flash interval includes zero, so these data show no noticeable loss from reuse. For Pro, the upper bound sits at zero while the lower bound drops almost 10 points. In the worst case, reuse costs noticeably more, and the benchmark makes that cost measurable.

Panel errors as a delivery indicator

The second dimension concerns execution and is built on 22 families. The authors looked at program assignments for which both saved initial states are delivered correctly, and searched for what distinguishes those cases from the rest. The distinguishing factor is panel correctness in the deployment state.

• Of 33 assignments with an error-free panel, 32 produced correct delivery in both states.
• Of 11 assignments where the panel contained errors, only one did.
• All program-gate regressions relative to the reference rules, and all cases of withheld delivery, fell into the group with panel errors.

The 97% vs. 9% gap, drawn from a small sample of 44 assignments, cannot be considered conclusive. But it suggests that testing a verifier on a test panel before launch gives a practical signal about whether the program can go into production.

A baseline for gates

The authors compared feedback strategies for the agent. Among the general-purpose variants tested, neutral continuation produced the highest share of correct delivery. In this setup, the agent simply keeps working with no gate intervening. This sets a floor: any verifier gate placed before delivery has to beat it, or it only adds complexity and the risk of false rejections. Web agents have not had such a floor until now, and teams often evaluated gates by eye.

What's in the release and how to use it

The benchmark is published together with instructions, environment states, verifier programs, reference rules and agent trajectories. The code and project page are available on GitHub and at ethanning.github.io/AgentProofBench. The preprint is registered as arXiv 2610.34634 and appeared in the Hugging Face Daily Papers digest.

For teams building web agents with automatic result checking, the benchmark suggests three concrete steps. First, measure how many generated verifiers survive a change in requirements, and do not rebuild them unnecessarily. Second, check the test panel for errors before deployment, since errors coincided with all regressions and withheld deliveries. Third, always compare a gate against the "no intervention" variant before concluding that it improves results. The limitations are also clear: four web services and two models from the same family give a narrow picture. But as a method for evaluation at the level of requirements and execution policy, the work provides a clear framework.