Alibaba DAMO Releases RynnUnify: A 16B Model That Plans in Video and Drives a Robot

Researchers at Alibaba DAMO Academy have released RynnUnify, a unified 16B-parameter model for embodied AI. It combines physical scene understanding, spatial grounding of objects, video generation and robot control in a single architecture. The headline numbers: +9.75 points over Cosmos 3 Nano on Rynn-VP, a long-horizon visual planning benchmark, 51.5% success on 20 composite RoboCasa365 tasks, and 46.7% on average across three tasks with a real robot.

One Brain Instead of a Stack of Three Models

Robotics stacks are usually assembled from separate parts. One VLM describes the scene, a second world model predicts what happens next, and a third policy turns all of that into motion. RynnUnify, built on Cosmos 3, tries to remove the seams between them. Inside is a Mixture-of-Transformers: an autoregressive Reasoner handles reasoning and understanding, while a rectified-flow video generator renders how the world will change after an action.

The key idea is called interleaved visual planning. The model alternates between text reasoning about a subtask and a short video forecast of its outcome, then uses that forecast to plan the next step. To keep the video from drifting over long horizons, the authors added three techniques: scene memory across video segments, self-forcing training and adaptive chunk duration.

An Agent That Decides When to "Imagine" the Outcome

Generating video is expensive, and not every question needs it. So the team trained an agentic layer on top of the base model using reinforcement learning (the BCPO method). The agent decides on its own whether to answer immediately or first generate visual evidence. If it takes the second path, it formulates a generation request, analyzes the predicted outcome and picks the reasoning branch that is more useful for the task. This matters most for unobservable states, such as what will be inside a box or how a pushed object will end up.

To evaluate the system, the authors built two new benchmarks: Rynn-VP for long-horizon visual planning and Rynn-GEU for anticipatory reasoning. On Rynn-GEU, agentic mode added +6.0 points to the average score compared with answering directly without video forecasts.

A Bridge from Prediction to Action

The third part of the system is the World-Action Model (WAM). It takes the full RynnUnify and trains it jointly on video and actions, turning the model's predictive representations of the world into robot commands. The experiments ran both in simulation and on real hardware.

• Model scale: 16B parameters, built on Cosmos 3.
• Architecture: Mixture-of-Transformers, with an autoregressive Reasoner and a rectified-flow video generator.
• Rynn-VP: overall score 9.75 points above Cosmos 3 Nano.
• Embodied scene understanding: the highest average score among the unified multimodal models evaluated.
• Rynn-GEU: the agentic layer adds +6.0 points to the average versus direct reasoning.
• RoboCasa365: 51.5% success on 20 selected composite tasks.
• Real robot: 46.7% average success on three tasks.
• New benchmarks: Rynn-VP and Rynn-GEU.

What Practitioners Should Do with This

The release is most interesting to teams building manipulators and mobile robots who are tired of stitching together a VLM, a world model and a policy by hand. The code and project page are open, so you can test how much a shared "brain" beats separate components on your own tasks. But the numbers are modest: 46.

7% on a real robot means more than half of attempts fail to reach the goal. What remains of confirmed value is the design itself, in which a single backbone understands, imagines and acts. Two things are worth watching: how much video generation costs at each planning step, and whether the results transfer to tasks outside RoboCasa365.

Independent reproductions will answer those questions. For now, the paper has collected 16 upvotes on Hugging Face Daily Papers, and the repository has only just started to gather stars.