AgentHorizon: Best Judge GPT-5.5 Scores Only 80.9% on Long Agent Tasks

The authors of AgentHorizon have released a benchmark of 1 373 tasks that tests not the computer-use agents themselves, but the automated judges that rule on their work. Of the eleven judges tested, the best was GPT-5.5 in an agentic configuration, with 80.9% balanced accuracy on the main AH split. Judges are already used both to train agents and to evaluate them without human involvement. Against that backdrop, the result on long tasks looks like a modest safety margin.

Why long trajectories are hard to judge

A computer-use agent's trajectory is a long sequence of screenshots and actions. From the outside it may look finished, yet still violate a constraint in the instruction or leave an unwanted side effect. For example, the agent completed the job, but did it in a different application or with different parameters. To catch this, the judge has to check the entire history against the user's original request. The evidence of success or failure is often buried somewhere in the middle of hundreds of steps.

The authors built the benchmark on 166 hours of human recordings across three operating systems. The key trick is how they obtained the negative examples. They recorded separate trajectories for instructions that are close in meaning, then swapped the instructions. As a result, the judge sees work that was done in good faith, but for a neighboring, incompatible request. A "looks reasonable" verdict does not work here: the judge has to verify that exactly what was asked for was done.

What was measured and how

The benchmark comes in three variants: the main frontier split AgentHorizon (AH), the simplified AgentHorizon-Simple (AH-S), and the developer split AgentHorizon-Development (AH-D). Eleven judges were tested in two ways. In the first, the full trajectory was passed to the model in one go. In the second, the models worked as coding agents inside five different agent harnesses and could explore the materials themselves.

• 1 373 tasks in an instruction–trajectory format
• 166 hours of human recordings, three operating systems
• up to 300 screenshots and actions in a single trajectory
• 11 judges, 5 agent harnesses
• best result: GPT-5.5, 80.9% balanced accuracy on AH

What the results show

Tool access does not help everyone. Some models did better with it, but open-weight models got worse. An agent harness does not deliver an automatic gain: the judge needs to search the data competently and not lose the thread over a long horizon.

The second observation concerns the nature of the errors. Judges differ sharply in their ability to accept a correct trajectory and reject a failed one. One tends to approve everything; another nitpicks correct solutions. Overall accuracy does not reveal this, which is why balanced accuracy is useful here: it averages quality across both classes, and random guessing in a binary setup gives about 50%. Measured against that baseline, 80.9% is a notable result, but it comes with a considerable tail of errors.

In their conclusions, the authors point to a shortage of judges that can find and verify the often hidden evidence of correct task completion within a long interaction history.

How to use this data in practice

Teams that train agents with an automated reward or compare them in evaluation runs can test their own judge on "own instruction – someone else's trajectory" pairs. The three AgentHorizon splits are enough for this. Look separately at the share of correct trajectories accepted and the share of incorrect ones rejected, not just at overall accuracy. For open-weight judges, check whether connecting tools breaks the result. While even the best judge errs noticeably more often than one would like, it makes sense to keep human spot-checking on long tasks.