RoboJev-27B judges robot success from 10 frames: 0.993 AUROC on SO-101

A huge share of robot manipulator videos are missing the one thing that matters most: a label saying whether the robot actually did what it was supposed to do. Josef Liyanjun Chen wants to close that gap with RoboJev-27B, an open model. It takes as input the task description, 10 frames from the attempt, and, if available, final frames from auxiliary cameras. The output is a probability of success and failure. On 268 held-out episodes from an SO-101 robot, it achieves 95.2% balanced accuracy and 0.993 AUROC.

A judge instead of manual labeling

The task looks modest, but a lot rides on it. Robot training datasets grow through labs and hobbyists writing teleoperation and autonomous runs. Without a success label, that material can't be reliably filtered, used for reinforcement learning, or used to compare policies against each other. Paying humans to watch hundreds of thousands of clips is expensive. Paying for closed API calls is expensive too.

RoboJev-27B is a variant of the JEV-27B-VL model. The author trained a rank-32 LoRA adapter on the language component, leaving the vision encoder frozen. The training data comprised 52,598 episode records. For research-sourced data, a KL penalty was added: it keeps predictions closer to the original model specifically on that data, so fine-tuning doesn't degrade what the model already knew how to do.

A separate finding is the calibrated two-token readout. The model doesn't produce a chain-of-thought — it outputs "success" and "failure" probabilities in a single forward pass. For large-scale evaluation this matters a lot: the threshold can be shifted to favor precision or recall as needed.

The numbers and their flip side

On the same robot used for training, everything looks great. On data from elsewhere, the picture is more modest. Metrics from the paper:

• 268 held-out SO-101 episodes: 95.2% balanced accuracy, 0.993 AUROC;
• 459 episodes from unfamiliar labs and other robots: 0.890 AUROC;
• 135 community episodes with unambiguous labels: 93.3% accuracy;
• the base model on a subset of unfamiliar labs and robots with the same inputs: 0.900 AUROC.

That last line matters. On data from "a different world," the fine-tuned version doesn't outperform the base model: 0.890 versus 0.900. All of the fine-tuning gain shows up where the robot and filming conditions are familiar. For a lab labeling recordings from its own platform, that's good news. For anyone hoping for a universal judge that works on any hardware, it's a reason to test the model on their own data first.

The author is also candid about the comparison with frontier VLMs accessed via API. Those models received only 4 images from the main camera, while RoboJev-27B got more frames plus additional camera angles. The conditions aren't equal, so this comparison can't be used to claim the open 27B model beats the closed giants. Part of the advantage comes from the input format itself.

Cost, and what you can take away

Economically, the model looks practical. The median time per episode is 0.26 seconds on a single H100 with sequential processing. At $2.69 per GPU-hour, that works out to roughly $0.19 per 1,000 predictions. This doesn't include media processing or serving costs, and in a real pipeline, video decoding could become a significant line item.

Along with the weights, the author has published the inference code and an evaluation log: 20,628 predictions grouped into 29 "model/protocol" records. These can be browsed and used for reproducible success-detection benchmarking. The project page is at robo-jev.com, and the preprint is arXiv 2610.33718.

The practical takeaway for teams collecting data on SO-101 or similar low-cost arms: RoboJev-27B can serve as a first-pass filter. It will weed out obvious failures and prioritize cases for manual review. For data from other platforms, it's worth first running a small labeled sample to confirm that an AUROC around 0.89 is acceptable. The probability threshold is best calibrated for your specific task — training set cleanliness matters more than recall.