Alibaba DAMO Trains Robots on 30,000 Hours of Video Without Action Labels

The biggest problem facing large vision-language-action (VLA) models for robots is a shortage of action-labeled data. Demonstrations with recorded commands are scarce, and every robot has its own control space. Researchers at Alibaba DAMO Academy propose a way around this constraint. Their RynnVLA-Latent framework pretrains on 30,100 hours of video featuring humans, robots, and simulations — none of which contain native control commands. The resulting 4-billion-parameter policy achieves a 98.8% success rate on LIBERO and 88.6% on the bimanual RoboTwin 2.0 benchmark.

Actions nobody recorded

At the core of the work is a model called RynnLAM. It looks at pairs of frames and learns to describe the transition between them as a continuous latent action. Because it doesn't need native robot commands, the method works with any video — including footage of human hands picking up objects.

This approach has a weak point: if the model simply predicts the next frame, it easily ends up encoding appearance changes instead — lighting, background, camera motion. The authors counter this with three techniques:

• residual future-feature prediction forces the model to encode the change between frames, not the frames themselves; • 3D-flow supervision with camera-motion compensation ties the representation to the actual movement of objects in space; • camera-motion decoupling separates camera movement from actions happening in the scene.

Together, these yield representations that capture motion relevant to control rather than just shifts in the image.

From 745 million tokens to a real robotic arm

The team applied RynnLAM to the full 30,100-hour corpus and obtained 745.52 million latent tokens. The policy itself is trained on these. Given visual observations and a language instruction, RynnVLA-Latent predicts chunks of latent actions using flow matching.

Next comes robot-specific post-training. The pretrained backbone and action expert module are adapted to the native control commands of a particular platform. Latent targets serve as a common language across different robot bodies, while platform-specific mechanics are added at the end.

The authors also tested the quality of the representations themselves. They probed frozen representations to show that they can predict control signals, ran ablations, and built scaling curves — quality improves as the volume of latent pretraining grows.

What the benchmarks showed

The 4-billion-parameter model delivered the following results:

• LIBERO — 98.8% success; • LIBERO-Plus — 84.7% with no benchmark-specific adaptation; • VLABench — 51.9% average across five tracks; • RoboTwin 2.0 (bimanual tasks) — 88.6% average; • six real-world manipulation tasks on platforms with a dexterous hand and a mobile gripper — the highest scores among the compared methods.

One caveat matters: in the real-world tasks, this is the best result among the methods the authors chose to compare against. The paper doesn't include a broad comparison with closed, industrial-grade systems.

A practical takeaway for the industry

The bottleneck in robotics today is data. Teleoperation is expensive, and dataset compatibility across different robots remains a chronic problem. DAMO's work points to a practical path forward: using accumulated video — including human footage — as a source of control signal.

For teams building manipulation models, this has several implications. First, unlabeled video can be turned into training targets if camera motion is properly removed. Second, a single pretraining run transfers across different platforms through brief post-training. Third, the scaling curve suggests that more video will keep improving results — and there's vastly more video in the world than labeled demonstrations.

This is an academic result, but the code has already been published in the alibaba-damo-academy/RynnVLA-Latent repository, and the project page is live. Independent teams will be able to verify the numbers and apply the method to their own platforms.