RoboJev-27B judges robot success from 10 frames: AUROC 0.993 on SO-101

A huge share of video from robot arms lacks the one thing that matters: a mark showing whether the robot did what it was supposed to do. Josef Liyanjun Chen proposes to close this gap with an open model, RoboJev-27B. It takes the task text, 10 frames of the attempt and, if available, the final frames from auxiliary cameras. It outputs probabilities of success and failure. On 268 held-out episodes from the SO-101 robot, it reaches 95.2% balanced accuracy and an AUROC of 0.993.

A judge instead of manual labeling

The task looks modest, but a lot depends on it. Robot-learning datasets keep growing thanks to labs and hobbyists who record teleoperation and autonomous runs. Without a success label, this material cannot be filtered reliably, used for reinforcement learning, or used to compare policies against each other. Paying people to watch hundreds of thousands of clips is expensive. So is paying for calls to closed APIs.

RoboJev-27B is a variant of JEV-27B-VL. The author trained a rank-32 LoRA adapter on the language part and left the vision encoder frozen. The training data consisted of 52 598 episode records. For research sources, a KL penalty was added: it keeps predictions closer to the original model on that particular data, so that fine-tuning does not damage what the model could already do.

Another notable feature is a calibrated two-token readout. The model does not write out any reasoning. Instead, it produces the probabilities of "success" and "failure" in a single forward pass. For large-scale evaluation this matters: the threshold can be moved to hit the desired precision or recall.

The numbers and their flip side

On the same robot used for training, everything looks excellent. On other people's data, the picture is more modest. Metrics from the paper:

• 268 held-out SO-101 episodes: balanced accuracy 95.2%, AUROC 0.993;
• 459 episodes from unfamiliar labs and other robots: AUROC 0.890;
• 135 community episodes with unambiguous labels: accuracy 93.3%;
• the base model on a subset of unfamiliar labs and robots with the same inputs: AUROC 0.900.

The last line is the important one. On data from "another world," the fine-tuned version does not beat the base model: 0.890 versus 0.900. All of the fine-tuning gain shows up where the robot and the recording conditions are familiar. For a lab labeling recordings from its own platform, that is good news. For anyone hoping for a universal judge for any hardware, it is a reason to test the model on their own data.

The author is also upfront about the comparison with frontier VLMs via API. Those models received 4 images from the main camera only, while RoboJev-27B got more frames and additional viewpoints. The conditions are not equal, so this comparison cannot show that an open 27B model is better than closed giants. Part of the advantage comes from the input format itself.

What it costs and what you can take

On economics, the model looks practical. The median is 0.26 seconds per episode on a single H100 with sequential processing. At $2.69 per GPU-hour, that comes to about $0.19 per 1000 predictions. The cost of media processing and serving is not included, and in a real pipeline, video decoding may become a noticeable line item.

Alongside the weights, the author published the inference code and an evaluation log: 20 628 predictions grouped into 29 "model/protocol" records. They can be browsed and used for reproducible benchmarking of success detection. The project page is robo-jev.com, and the preprint is arXiv 2610.33718.

The practical takeaway for teams collecting data on SO-101 or similar low-cost arms: RoboJev-27B can serve as a first-pass filter. It will weed out obvious failures and prioritize what to check by hand. For data from other platforms, first run a small labeled sample and make sure an AUROC of around 0.89 is acceptable. It is better to calibrate the probability threshold to your own task: the cleanliness of the training set matters more than recall.