LeWorldModel Tested for Action Sensitivity: Axes Diverge by 36–42°

A pretrained world model can predict the future well and still respond incorrectly to small changes in actions. That is what Vasil Vasilev and Siddharth Nayak tested in arXiv preprint 2610.31716 on LeWorldModel (LeWM). On nearly a thousand states from the Reacher and OGBench Cube environments, the most sensitive action directions in the model and in the simulator diverge by a median of 36° and 42°, respectively. This result is for the smallest of the three perturbation scales.

How action sensitivity was measured

A model that predicts the next observation does not necessarily understand how the response depends on the action. For planning and control, the second matters more. A planner iterates over candidate actions and picks the one after which the predicted future looks best. If the model misrepresents which movements change the outcome most, the optimization runs over a distorted landscape.

The authors built Gram matrices for actions, that is, a description of how strongly and in which directions a small change in action shifts the outcome. To compare the two, they used paired counterfactual simulator runs and the model's responses from the same state. They measured sensitivity at three perturbation scales and in three representations: in pixels, in embeddings, and in selected physical state coordinates.

What Reacher and Cube showed

The main result concerns directions. When some combination of action components changes the world most in the simulator, LeWM, at the median, treats a different direction as the most sensitive one, deviating by 36° in Reacher and 42° in OGBench Cube. This is about the principal axes of sensitivity, not random noise at individual points.

Scale matters a lot. When the action perturbation is increased tenfold, the median normalized divergence between Gram matrices drops by about 70% in Reacher and 47% in Cube. The model is closer to the environment on coarser action changes and diverges more on fine ones. Any conclusion about LeWM's quality therefore depends on the scale chosen for the test.

Pixels, embeddings, and physics

The measured geometry also depends on the space in which it is computed. In pixels and embeddings, it changes between scales noticeably more than in physical state coordinates. The assessment of the model comes out differently depending on whether you look at the image, the latent representation, or the position and velocity of objects.

Separately, the authors tested response prediction on held-out directions that were not used to build the metric. The results, in median errors of the embedding response:

• Reacher, environment metric (environment Gram): 17%
• OGBench Cube, environment metric: 28%
• Reacher, isotropic reference with the same trace: 26%
• OGBench Cube, isotropic reference with the same trace: 38%
• LeWM at this scale: higher errors than the environment metric

The isotropic reference spreads sensitivity evenly across all directions and preserves the overall response strength, i.e. the trace of the matrix. It serves as a lower bound: the environment metric beats it, while LeWM's geometry gives less accurate predictions of response magnitude on new directions than a metric derived from the environment.

What to check before planning with a world model

The authors' practical takeaway: evaluating a world model by prediction accuracy alone is not enough. You need to look at how predictions change when actions change, and do so at several perturbation scales and in several representations. If the check runs at only one scale, it may make the model look better or worse than it will behave in a real task.

The work does not prove a link between these divergences and control quality. The authors explicitly call the impact on control an open question and ask colleagues to share their views on the implications for model-based planning. The study covers two environments and one pretrained model, so it is too early to generalize it to all world models. Still, teams building planners on top of latent models now have a concrete set of diagnostics: Gram matrices, counterfactual simulator runs, and held-out directions. The project is posted at geometry.vasi.boo.