RoboCasa-Shift: a benchmark shows where VLA robots break in the kitchen

The same drop in average success rate can mean entirely different robot failures. That is the conclusion of the team behind RoboCasa-Shift, a diagnostic benchmark for VLA (vision-language-action) policies built on the RoboCasa365 simulator. The authors ran GR00T N1.5 through it on four long kitchen tasks, four perturbation factors and three calibrated severity levels. The result is 1 040 scored rollouts, including conditions under which the policy failed the task completely.

One example shows why this matters. A camera shift at level L2 and Gaussian blur at level L1 produce the same drop in success rate, ΔSR=0.188. By the aggregate metric, they are twins. But a stage-by-stage breakdown shows that the policy breaks at the first stage of the task in 19.0% of cases under the camera shift and in 36.4% under blur. The headline number is the same, but the failure trajectories differ.

Paired rollouts instead of comparing averages

VLA robustness is usually evaluated like this: take a clean run, take a perturbed run, and compare the average success rate. The problem is that the two sets of episodes differ not only in the target shift but also in random initial conditions. RoboCasa-Shift is stricter. Each pair of rollouts uses the same base scene settings and differs only in the perturbation under study. The shifts are also single-factor: exactly one factor changes at a time, so there is no need to guess what caused the degradation.

This design exposes an effect that averages hide. One of the camera conditions produced ΔSR=0, meaning that formally nothing changed. Yet in 8 of 80 paired episodes, the outcome did change. Some tasks the robot used to complete began to fail, while others, conversely, it suddenly completed. The average simply netted out the gains and losses. For deployment, a zero delta like this is dangerous: it creates an impression of robustness where the policy's behavior is in fact unstable.

Depth of failure as a separate metric

The third element of the benchmark is stage-level diagnostics tied to the simulator state. A long household task consists of sequential stages, and the authors track how far the policy gets before failing. This yields an execution-depth profile: a robot may stumble right at the grasp, or it may get almost to the end and drop the result on the final step. For an engineer, these are fundamentally different bugs with different fixes. The first often points to perception problems, the second to error accumulation over the course of the task.

Another observation: increasing perturbation severity does not always lead to monotonic degradation. A stronger shift can turn out to be less harmful to the policy than a weaker one. It follows that a single "perturbation strength vs. quality" curve is not enough for VLA models.

Study parameters

• Benchmark: RoboCasa-Shift, built on RoboCasa365
• Policy evaluated: GR00T N1.5
• Tasks: four long household scenarios that passed a predefined quality threshold under clean conditions
• Perturbation factors: four, each with three calibrated severity levels
• Scale: 1 040 scored rollouts
• Metrics: success-rate drop (ΔSR), outcome sensitivity in paired episodes, share of failures at each stage
• Code: open-sourced on GitHub, repository HaochengWang23/RoboCasa-Shift

How to change the way robots are tested

For teams that fine-tune or deploy VLA models, the takeaway is practical. A single success-rate percentage is not enough. Reporting needs at least three layers: overall degradation, the number of episodes where the outcome changed despite a zero average delta, and the distribution of failures across task stages. The first layer answers "how bad is it," the second "how unstable is it," and the third "where do we fix it."

The task selection criterion matters too. The authors admit to the analysis only tasks that the policy solves reliably without perturbations. Otherwise, it is impossible to tell the model's fragility apart from its inability to cope even in a clean environment. This practice is worth adopting when comparing different models as well.

The limitations are also clear: one policy and four tasks were evaluated, and only in simulation. How these failure profiles carry over to real kitchens and other VLA models will be shown by future work. But the community now has an open tool for asking such questions under identical conditions. For robotics, where a mistake can mean broken dishes or a stopped production line, this is a significant step from leaderboards toward diagnostics.