World models: will they be the key to autopilot?

Over the past couple of years, automakers talking about intelligent driving have invariably dropped new technical terms. After end-to-end learning and VLA, the "world model" is the trendiest phrase in intelligent driving. Different companies have even given it their own spin: Xiaopeng introduced a "World Foundation Model", NIO called it an "End-to-End World Model", and Huawei a "World Behavior Model" (WA). Horizon Robotics, Li Auto, Yuanrong Qixing and Momenta are also working on world models.

But judging by their press conferences, it is hard to tell whether the world model they are talking about is the same thing. What problem does it solve, and where does it sit in the intelligent driving architecture? In the broader context, a "world model" is essentially a re-creation of the real world in a virtual one: a technology that lets AI understand the real world, learn physical laws, cause-and-effect relationships and the dynamics of the environment, much as a human does.

Most scientists and technology companies see world models as a key element of "physical-world AI". Stanford University professor Fei-Fei Li has said that spatial intelligence is the next decade of AI, and that the world model is the key technology for building it. Researchers and technology companies at the leading edge are still in the exploratory stage, but China's auto industry has already staked out positions with a range of new conceptual terms.

In fact, the "world model" discussed today in the intelligent driving industry is merely a difference in naming, with no big difference in technology. It is an upgrade to the industry's original simulation tools. It tackles the testing and verification of end-to-end models in a virtual world with higher fidelity, finer detail, richer scenes and more degrees of freedom. All of this serves the goal of training a more capable, more human-like end-to-end intelligent driving model.

In other words, intelligent driving suppliers and automakers are not actually building a complete digital physical reality. They are simply using the idea of a world model to build a simulator. Each company may have different expectations of the world model, but as far as we know, world models in the intelligent driving industry are currently used only in the cloud and are not used in vehicles.

The spread of end-to-end learning has exposed the weaknesses of simulators. Over the past two to three years, leading intelligent driving solutions have moved from a rule-based stack to AI-based control and completed a "formal" unification. Perception, prediction and planning have been integrated into a single network as far as possible, combined with larger models and more computing power. As automakers often say at their press conferences, "intelligent driving after end-to-end learning feels more like human driving".

In real-world use, however, a counterintuitive phenomenon appeared: new OTA versions after end-to-end learning do not necessarily get better and may even "regress". The root problem is not that the model got worse, but that AI-based control makes evaluation and regression testing harder. At the time, many intelligent driving specialists believed that as long as the front end was trained well enough, the car would drive like a human.

This path is not fruitless, and the early results of end-to-end learning amazed many specialists, but the "black box" of end-to-end learning also has side effects. When the model makes a mistake, it is hard for developers to find out why. How can they prove it will not happen again next time?

Whether a model is good depends not only on whether it is "big enough and has enough data", but also on how you detect problems, define them and verify them. Manufacturers gradually realized that they needed a better simulator to evaluate model performance at the model verification stage.

Most leading players are building world models to use as simulators. To let an ideal VLA run reinforcement learning in a simulated environment, Li Auto proposed a world driving model in 2025 that includes the trajectories of both the ego vehicle and other vehicles and acts as an evaluation teacher. Xiaopeng has only announced a "World Foundation Model", which is essentially unrelated to a world model, but according to 36Kr Auto, Xiaopeng also uses a world model for simulation and for testing to evaluate the algorithm capabilities of new model versions.

The spread of end-to-end learning has exposed the weaknesses of traditional simulators. "When end-to-end learning was not so popular, verification was not so costly for everyone, and they could still verify the system piece by piece. Now that end-to-end learning is here, there is no way to verify the system piece by piece, and the simulator problem becomes obvious," said an industry developer.

In the rules era, the simulations automakers built often served two purposes. One was replaying takeover problems and reproducing fragments that occurred during road tests. The other was using simulators to increase the richness of corner-case data, creating several typical intersections, pedestrians crossing the road and vehicle cut-in scenarios in the simulator so that the system could work through them. At that time, the simulator played the role of a "magnifying glass". After end-to-end learning, however, the model is hard to split into parts, it is hard to systematically generate smaller, controllable corner cases, and it is even harder to support the large-scale closed-loop verification that end-to-end learning requires. This is exactly why the world model was introduced.

In the end-to-end era, the world model is the "coach" of the intelligent driving model. "Currently, the level of domestic automakers' world models is at some distance from Tesla, but the gap is less than a year," said an industry insider.

Tesla did not use the concept of a "world model" but the term "world simulator" (Tesla's vice president of autonomous driving, Ashok Elluswamy, first mentioned it at last year's ICCV). The simulator is built on a huge dataset that Tesla created itself, and it generates a future state based on the current state and the next actions. It is thus closed-loop with the base end-to-end model on the vehicle side to evaluate real-world performance.

The industry insider noted that Tesla is more like using neural networks to "fit" the world. The rendering process is generated through computation to minimize the explicit imposition of physical rules; the materials library is not fully predefined by humans in advance, but retains a certain probability weight and combinatorial space. The advantage of this approach is that the model has stronger generalization ability.

Domestic automakers are taking a different, more "controlled" path. According to a supplier who spoke with 36Kr Auto, Li Auto uses 3D Gaussian reconstruction, which is also one of the methods most automakers currently use.

Whichever route is chosen, the world model ultimately points to the same position in engineering terms: automakers use it as a "verification and refutation system" in the end-to-end era. In the cloud, it reproduces, rewrites and extends situations that may occur in real driving, checks whether the output of the large model on the vehicle side is stable and reproducible, and turns "where it went wrong and why" back into a traceable chain of evidence.

The world model's role is like that of a coach, and an excellent coach can train excellent athletes. "As the cloud world model gets stronger and stronger, in theory the end-to-end model trained on the vehicle side should get stronger and stronger," said a developer.

The core capabilities of a world model mainly cover two aspects. One is digital modeling and abstraction of the physical world. The other is reasonable imagination and prediction of the physical world based on that modeling, for example predicting how the future world will change from given images. Whether a world model is good depends on whether it can generate sufficiently realistic and diverse data in the cloud. "If an automaker only uses collected real data for simulation, it is clearly not building a world model, just a data replay pipeline," said a supplier's product manager.

A world model has to learn how the world operates from data of the physical world, so the quality of its training data will substantially affect the quality of what the model generates. Mao Zimin, head of the product line at JIJIA Vision, said: "For a generative model such as a world model, its generated output will ultimately follow the distribution patterns of the features in its input data. In the course of commercializing a real world model, we found that if the data quality is only 60 points, the data generated from that world model may be only 55 points."

Based on a world model, automakers can generate the scenarios they need without limit across various dimensions in cloud simulation, and can generate videos as training data according to instructions. "The efficiency is not just slightly higher than collecting real data and then training; the model iteration speed will lead the era," said a supplier's developer.

But these are all idealized results. "The world model is a big upgrade over the simulators used for intelligent driving, or in other words over the lack of simulation information, where verification could only use offline data, but it is still far from an ideal simulator."

The world model algorithm is not yet mature and still has many "hallucinations". The industry as a whole is currently at the "just getting started" stage.

An automaker developer told 36Kr Auto that domestic manufacturers can generate video clips of 30-60 seconds based on a world model, but the consistency of dynamic objects is not very good, and there are major problems with both spatiotemporal consistency and multi-view consistency.

The core of a world model is a generative model, and a generative model inherently carries the risk of "hallucinations". "The hardest part of the world model right now is how to guarantee that what is generated is real. If a person is generated, how do you guarantee that their behavior and trajectory could happen in the real world," said a supplier's product manager. "If the world model produces confusion, the model will learn the wrong things, which will lead to a very bad result for the model deployed on the vehicle side."

An extreme example: if cars generated in the cloud move sideways, the model will assume that a car at front left will instantly move to front right. During actual driving, the model may brake.

If a simulator cannot approximate key real-world causal relationships, such as the effect of a slippery road on braking distance, the probability of false detection of stationary objects in backlight, or the bargaining strategy of an oncoming vehicle during a lane change, then the "corner case" it generates may be false. Optimizing against false problems amounts to spending development resources on phantoms.

Many believe the bottleneck of world models is data and computing power, but Xia Zhongpu, former head of Li Auto's end-to-end autonomous driving model, agrees more with LeCun's view: "There are no big breakthroughs in world model algorithms, and self-supervised learning on image models has not yet found a relatively smooth paradigm the way language has."

Language models can scale quickly because language itself has high information density, and each word carries clear semantic constraints. Image information density is low, and for a "driving decision" only a small part of the information is useful.

For example, the model does not need to predict the trajectory of a car far behind, or changes in distant buildings; that is all noise. But it must predict whether the car ahead in the same lane will suddenly brake hard, whether a car in the adjacent lane is about to change lanes, and whether a pedestrian is about to suddenly cross the road. The model must first know "what to pay attention to".

"At present, intelligent driving algorithms cannot extract enough useful image information for driving," said Xia Zhongpu. An image may contain millions of pixels, but only 20 or so pixels are relevant to decision-making, and the rest is noise. The model must first learn to extract 1‰ or even 1‱ of effective signal from the noise, and only then can we talk about organizing the signal into a structure that can be used for reasoning and prediction.

In Xia Zhongpu's view, the world model algorithm has not yet broken through, let alone the questions of whether there is enough data and how much computing power is needed. Precisely because the underlying technology of the world model has not yet seen a clear breakthrough, automakers' investment is more exploratory in nature, and even some automaker bosses are confused by it.

If a world model is made good enough, it could be placed on the vehicle side, provided computing power can support it. "Currently in China, the world model is mainly used as a simulation system, and the understanding of decision-making technology for intelligent driving is still not high enough," said Xia Zhongpu.

This also explains an apparent contradiction: why everyone talks about world models, yet the difference in user experience is not obvious. Most people's world models are still at the first stage, "used for training and verification", not the second stage, "able to support planning and decisions".

"Deploying a world model on the vehicle side is the hardest part," said Xia Zhongpu. At present, no company applies a world model on the vehicle side. He also noted: "Use large-model methods to simulate the physical world, predict how the world will develop through interaction with it, and therefore influence the world through decisions so that it develops in a direction favorable to oneself. If world models reach that level, the problems of autonomous driving and robotics can be solved."