Survey by Shengqiong Wu and Hao Fei: 600+ Papers on Video Models as World Models

Over the past few years, video models have developed along three parallel tracks: some learned to understand clips, others to generate them, and still others to predict the dynamics of the world. Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr and Hao Fei have pulled these tracks together into a single survey. It covers more than 600 papers, organized into 19 tables, and poses one question common to all the directions: what connection does a model learn between the video it observes and the world that produced that video?

Video as a fragment of the world

The starting metaphor is simple. A video is a partial, lossy, time-ordered trace of a world that keeps changing even when the camera is not looking. A car drives behind a wall and the frame is empty, but the car still exists in the world. A model that fails to account for this can produce a beautiful picture while having no real representation of what is happening.

This leads to the survey's key distinction. Temporal coherence, meaning the smoothness of frames, is a property of the trace itself. World consistency is a property of what that trace implies. One does not follow from the other, yet most current metrics check only coherence.

Four operations instead of three communities

The authors sort the literature into four complementary operations, each with its own definition and evaluation criteria:

• Reading: the model infers the state of the world from video (understanding, recognition, recovery of scene properties).
• Writing: the model synthesizes video from given or learned conditions (generation).
• Sharing: a single model both reads and writes, with understanding and generation joined through shared modeling mechanisms.
• Interacting: the model predicts and updates the state of the world in response to actions and changing observations.

The survey also offers a taxonomy of video world models along two axes. The first axis is the prediction space: observations or a latent state. The second is the transition interface: passive, action-conditioned or interactive. Sora, V-JEPA and Genie 3 each occupy their own cell in this grid, and no single model covers it entirely.

The evidence ladder and seven unsolved problems

The authors propose four rungs for judging claims of a "world model":

1. plausible output; 2. fidelity to the given conditions; 3. a consistent world state; 4. usefulness for decision-making.

In the authors' view, a system needs at least the third rung to be called a world model. A plausible video, or even precise adherence to a prompt, is no substitute.

The survey formulates seven open problems, each with a direction for methods and an evaluation protocol:

• long-horizon consistency;
• physical and causal grounding;
• a measurable synergy from combining understanding and generation;
• standardized evaluation;
• control learned from unlabeled video;
• generalization to new scenes and embodiments;
• stochastic worlds and rare events.

The survey traces an overall shift from clip-level recognition and perceptual fidelity toward a persistent state, controllable scene evolution, physical and causal correctness, and evaluation through interaction and downstream decisions. As the authors stress, scaling world models requires interventions, not simply more videos: observation alone does not show what would happen under a different action.

How to use it

The survey works as a checklist when reading the next video-model release. If an announcement offers only polished demos and FVD-like metrics, you are looking at the first or second rung of the ladder. Questions worth asking developers: does an object persist off-screen, does the scene state change under an agent's actions, and has the model been tested on decisions in downstream tasks such as robotics or planning.

For a team choosing a research or investment direction, the seven problems provide a ready-made map of gaps. Long-term memory, causality and learning control from unlabeled video remain the weakest areas, and that is where the least work has been done so far.

The survey's project page is vid-gen.github.io/video-world-model-survey, and the list of papers is maintained in the vid-gen/awesome-video-world-model repository on GitHub. The authors ask readers to submit any papers they missed. The paper was released on 22 October 2026 and appeared in the daily Hugging Face Papers selection, where it collected 218 upvotes.