Yann LeCun unveils LeWorldModel, a JEPA model that trains on pixels without representation collapse

A research team led by Yann LeCun has introduced LeWorldModel, or LeWM, a new world model for training agents directly on pixel data. The authors claim the model solves one of the main problems of the JEPA approach, representation collapse, while also speeding up planning noticeably.

Why JEPA breaks

Agents need world models to build a compact internal map of their environment and to evaluate actions in a latent space rather than in raw frames. But when trained directly on images, such systems often fall into representation collapse: different scenes start to be encoded too similarly, and the model formally completes the prediction task while losing the useful structure of the world. As a result, developers have to safeguard training with auxiliary tricks such as stop-gradient, EMA, frozen encoders and multi-term loss functions.

The problem is especially painful for agents that must plan long chains of actions: if the latent space degenerates, the planner can no longer tell good scenarios from bad ones.

How LeWM works

LeWM tries to remove this complexity. The architecture has two main parts: an encoder that converts a frame into a compact latent representation, and a predictor that estimates the next state from the current state and an action. The implementation uses a ViT-Tiny of roughly 5 million parameters and a transformer predictor of roughly 10 million, so the whole system fits in 15 million parameters and, according to the authors, trains on a single GPU in a few hours.

The key idea is to avoid piling on auxiliary objectives and keep only next-embedding prediction and a SIGReg regularizer. SIGReg pushes latent vectors to stay diverse and close to an isotropic Gaussian distribution. To do this, the model does not look at the whole space at once; it examines a set of random one-dimensional projections and checks their statistics. This approach should reduce the risk of degenerate representations without heavy engineering scaffolding.

In practice, LeWM is left with one truly important hyperparameter, the regularization weight λ, whereas PLDM, its closest end-to-end alternative, has substantially more such settings. The authors also note separately that a dropout of 0.1 in the predictor and a small projection layer after the encoder helped with stability.

What the tests showed

According to the paper, LeWM turned out to be not just more stable during training but also faster at planning. The authors compare it with PLDM and DINO-WM on navigation, manipulation and control tasks in 2D and 3D environments. The model works directly from pixels, without a frozen foundation encoder and without being tied to tasks that require a reward, yet it remains competitive on several benchmarks.

• about 200 times fewer tokens per frame than DINO-WM
• up to 48 times faster planning: roughly 0.98 seconds versus 47 seconds per cycle
• only two loss functions instead of seven in VICReg-based PLDM approaches
• one main hyperparameter instead of a set of hand-tuned settings
• the latent space captures physical quantities and flags "impossible" events such as objects teleporting

The authors also tested separately whether the model understands the physical logic of a scene rather than merely guessing the next frames. In violation-of-expectation tests, the system reacted more strongly to physically implausible events, such as a sudden teleportation of an object, than to purely visual changes.

Another interesting effect is temporal latent path straightening: as training progresses, trajectories in the latent space became smoother and more linear, even without a separate penalty designed to force that behavior. This matters because smoother latent trajectories usually make it easier to search for actions during planning.

What this means

For the agent market this is an important signal: world models are becoming a practical direction again, not just an academic idea. If LeWM and similar approaches confirm their results beyond laboratory benchmarks, developers could build faster, cheaper agents that plan in a compact state space without heavy foundation encoders. This is especially interesting for robotics, offline RL and systems where the cost of errors and latency is critical.

In essence, LeWM shows that collapse of representations can be tackled not by complicating the stack, but by framing the learning problem itself more carefully.