ReplayRCA: Forcing a Last-Step Fix Costs a Strong Model 17.1 Points of Causal Fix

When an LLM-based agent fails a multi-step task, another model is increasingly asked to work out why. The authors of the ReplayRCA benchmark propose checking such a diagnosis by its consequences, not by matching it against human labels. The environment is returned to its state before the chosen step, the agent's action is replaced with the one the diagnostician proposes, and the same agent then carries on by itself.

The main metric is the causal fix: the share of successful continuations after the edit minus the share of successful continuations in a control run that replays the agent's original action (a paired placebo). The headline result is measured in these same points: if a strong model is forced to intervene only at the last step, it loses 17.1 points of causal fix.

How to separate diagnosis from luck

The placebo exists so that the diagnostician is not credited for something the agent would have achieved anyway on a second attempt. The score subtracts the chance of recovering unaided, the effect of a restart and sampling noise. Neither a judge model nor annotators take part in the scoring: the replay itself is the ground truth.

There is a reason for this approach. Existing failure-attribution benchmarks compare an answer with a label, and the label is debatable: the "decisive step" is an annotator's judgment, and a different but equally successful fix point counts as an error. In addition, according to a paper by Naagarajan and Streif (2026), the answer "always the last step" is correct in 67–96% of cases on existing datasets, and falls to nearly chance level only on failures whose decisive step is confirmed by replay.

The tasks come from OpsWorld, a deterministic, procedurally generated back-office environment with 15 tools, three task families and six difficulty levels. The top level is multimodal: QR codes, photos of forms and handwritten text. Instances are created from a seed, so fresh tasks can be generated anew, which limits memorization of published ones, and every state restoration is verified.

What 13 models showed

On 62 calibrated system failures, Grok 4.7 posts the best result. However, none of the 78 pairwise differences survives correction for multiple comparisons, so the observed order cannot be treated as a reliable hierarchy. The flagships of the two largest vendors, GPT-6 Astra and Claude Fable 5.1, landed in 8th and 12th place. The authors note a caveat: the Claude models were run below the maximum reasoning level, and an ablation shows this costs Claude Haiku 5.5 15.2 points. No model fixed five of the tasks.

The structural trick "always the last step," which dominates label-based benchmarks, does not carry over here: forcing an intervention at the last step costs a strong model the 17.1 points mentioned above.

Explaining a failure and preventing it are different skills

The second track is live oversight. A supervisor watches a run as it unfolds and can intervene once, before an irreversible action, without knowing whether the run will fail. Successful runs are mixed into the sample so that false alarms can be measured. This is a scalable-oversight and AI-control setup, but applied to honest mistakes rather than adversarial agents, and scored by consequences.

On 87 runs, the two best supervisors are statistically indistinguishable, and one of them is a small open model, DeepSeek V4.1 Flash. GPT-6.1 Sol is among the best post-hoc diagnosticians but turned out to be the weakest as a live supervisor. It has the largest gap between "explaining" and "preventing" on the same runs. The authors conclude that the quality of post-failure analysis may be a poor predictor of real-time control quality.

Limitations and novelty

Counterfactual replay is not new in itself: CausalFlow, REFLECT, SymTrace and other works use it, but as a way for a model to check its own decisions. In ReplayRCA, replay is the scoring rule for any diagnostician model, each score is computed relative to a placebo at the same step, and the same failures serve both post-hoc analysis and live oversight.

The published instances and the generator's structure remain "learnable." The authors discuss regenerating the pool as agents improve but have not yet been able to test it. They present ReplayRCA as a reusable protocol rather than a leaderboard: the environment and the scoring code are published on GitHub. The practical takeaway for teams building agent oversight: choosing a supervisor by the quality of its post-hoc analyses is risky, and it should instead be tested on live runs with a mix of successful and failed launches.