Alibaba DAMO trains robots on 30 thousand hours of video without action labels

The main bottleneck for large robot models (VLA, vision-language-action) is a shortage of data with action labels. Demonstrations with recorded commands are scarce, and every robot has its own control space. Researchers at Alibaba DAMO Academy propose a way around this limit. The RynnVLA-Latent framework pretrains on 30.1 thousand hours of video of humans, robots and simulations that contain no native control commands. The resulting 4 billion-parameter policy reaches a 98.8% success rate on LIBERO and 88.6% on the bimanual RoboTwin 2.0 benchmark.

Actions that nobody recorded

The work is built on a model called RynnLAM. It looks at pairs of frames and learns to describe the transition between them as a continuous latent action. It does not need native robot commands, so any video works, including clips of human hands assembling objects.

This approach has a weak spot. If the model simply predicts the future image, it easily starts encoding changes in appearance: lighting, background, camera movement. The authors counter this with three techniques:

• residual future-feature prediction forces the model to encode the change between frames specifically; • 3D-flow supervision with camera-motion compensation ties the representation to the actual movement of objects in space; • camera-motion decoupling separates camera movement from actions in the scene.

Together, these produce representations that describe motion relevant to control, not just a change of picture.

From 745 million descriptors to a real hand

The team applied RynnLAM to the entire 30.1 thousand-hour corpus and obtained 745.52 million latent descriptors. The policy itself is trained on them. Given visual observations and a language instruction, RynnVLA-Latent predicts latent-action chunks using flow matching.

Post-training for a specific robot comes next. The pretrained backbone and the action expert module adapt to the native control commands of the particular platform. The latent targets serve as a shared language across different embodiments, and the specifics of a given mechanism are added at the end.

The authors also checked the quality of the representations themselves. They tested the frozen representations with probes to show that control can be predicted from them. They also ran representation ablations and built scaling curves: quality grows with the volume of latent pretraining.

What the benchmarks show

The 4 billion-parameter model produced the following results:

• LIBERO — 98.8% success; • LIBERO-Plus — 84.7% without adaptation to this benchmark; • VLABench — 51.9% on average across five tracks; • RoboTwin 2.0 (bimanual tasks) — 88.6% on average; • six real-world manipulation tasks on platforms with a dexterous hand and a mobile gripper — the highest scores among the compared methods.

One caveat matters: in the real-world tasks, this is the best result among the methods the authors chose to include in the comparison. The paper does not offer a broad comparison with closed industrial systems.

What this means for the industry

Data is the bottleneck in robotics today. Teleoperation is expensive, and dataset compatibility between robots remains a chronic problem. The DAMO work shows a practical path: use accumulated video, including human video, as a source of control signal.

For teams building manipulation models, this means a few things. First, unlabeled video can be turned into training targets if camera motion is removed correctly. Second, a single pretraining run transfers to different platforms through short post-training. Third, the scaling curve suggests that even more video will yield even better results, and there is far more video in the world than labeled demonstrations.

This is an academic result, but the code is already published in the alibaba-damo-academy/RynnVLA-Latent repository, and the project page is open. Independent groups can verify the numbers and apply the method to their own platforms.