RoboValue Tests Robot Action Evaluators on 35 Tasks and Finds Blind Spots

Researchers have published RoboValue, a benchmark for evaluating value models in robotics on a common footing. These are systems that look at frames of a task being carried out, read the instruction, and output a score: how close the robot is to the goal and whether it is moving in the right direction. The dataset covers 15 simulated and 20 real-world manipulation tasks. The main result is unwelcome for developers: models that score well on task outcome and forward progress still make systematic errors in the finer details.

Why test the evaluator, not just the robot

A value model serves two functions. First, it provides feedback during policy training: its signal tells the robot which actions were useful. Second, it monitors execution in real time and can flag that something is going wrong. If the evaluator is wrong, the policy learns from a distorted signal, and monitoring either misses failures or raises false alarms.

According to the authors, existing benchmarks cover only individual aspects of execution understanding. The reliability of value feedback as a whole remains insufficiently tested. RoboValue fills this gap with a single set of interfaces through which different models can be run.

Four axes and the dataset's composition

The framework evaluates four complementary capabilities:

• understanding of task state: what has already been done and what remains;
• monitoring of temporal progress: whether the score rises as the task advances;
• reasoning about failures and recovery: whether the model notices a mistake and recognizes a correction;
• value consistency: whether scores agree with one another and with the task's requirements.

The benchmark is built on diagnostic trajectories and instruction variations. They test whether the score reflects the task's actual requirements and whether it changes as execution progresses. RoboValue-Dataset contains expert demonstrations for training and annotated trajectories for detailed analysis. The models were tested in zero-shot and one-shot modes, under standard conditions and separately under changes of embodiment (robot type) and environment.

Where value models go wrong

The authors identify three groups of problems. The first is fine-grained grounding of the instruction in the observed scene. When the wording changes in its details, the score does not always follow the meaning. The second is tracking history-dependent progress. A model that has to remember what has already happened gets confused where a single frame is not enough. The third is judging recovery after a failure. This is where the models do worst, even though they look confident on outcome and forward-progress metrics.

A separate observation concerns the order of actions. The same semantic subtask receives a substantially different value gain depending on which of the valid orderings the robot uses for its steps. For a task where several sequences are equally correct, this means a noisy and biased signal.

What teams training robots should do

The practical takeaway is that a single success metric on the final outcome is not enough to trust a value model as a source of reward. Before using such a model in RL or in monitoring, teams should run it through checks for rephrased instructions, for trajectories with failures and corrections, and for alternative step orders.

The authors call the errors they found concrete targets for value supervision. Training data should be collected so that it includes recovery attempts and different valid sequences, not just clean successful demonstrations. The public leaderboard and the project site at robovalue-benchmark.github.io let teams compare their own models with others and track whether these weak spots are being closed.