Researchers Explain How JEPA Selects Features: Relationships Matter, Not Lone Features

Researchers Mosh Levy, Aly Lidayan and Yoav Goldberg have published a paper that changes the explanation of what ends up in a JEPA (joint-embedding predictive architecture) representation. It was previously assumed that the architecture keeps features that are well predictable on their own. The authors argue that this picture is incomplete. Selection happens at the level of feature sets whose members predict each other. They named this property predictive closure. The paper, 2610.27364, appeared on 19 October 2026. Mosh Levy presented it on Hugging Face Daily Papers, and the code is in the repository Mosh0110/predictive-closure.

Why the traffic light should not disappear

In JEPA, the encoder and the predictor are trained together. The task is to minimize the error in predicting the representation of the next observation. There is a non-obvious detail here: the representations of both the current and the next observation are learned, not fixed in advance. So it is unclear beforehand which features will end up in them.

Prior work described selection through the individual predictability of each feature. That led to a concern: the encoder would simply discard a feature that is unpredictable on its own. The authors use a traffic light as an example. It is hard to tell when it will switch. But its color is a strong help in predicting whether traffic will stop. Prediction error depends not only on the features themselves but also on the links between them, and older descriptions did not account for those links.

Selection at the level of sets

The authors reformulated the JEPA training objective and showed that the outcome is determined by properties of feature sets, not of individual elements. If features evolve independently of each other, the new characterization matches the earlier ones. The two diverge when features are linked. In that case, an unpredictable feature like the traffic light can stay in the representation because it is needed to predict other features.

The theory was tested in controlled experiments. The results:

• unpredictable features are represented better when they help predict other features;
• changing the predictive links of one feature affects how well the others are represented;
• an auxiliary objective targeting one feature strengthens the representation of another feature whose future values the first helps predict;
• feeding a feature to the predictor as side information produces the same indirect effect.

Indirect control of the representation

In practical terms, the most interesting finding concerns controllability. If you want a model to encode a particular feature better, you do not have to act on it directly. You can strengthen a feature linked to it, either by adding an auxiliary objective for that feature or by passing it to the predictor as additional information. The target feature's representation improves if the strengthened feature helps predict its future. It also works in reverse: if a feature's links are weakened, other features that relied on it may suffer.

The main limitation of the work is scale. The experiments are controlled, meaning they use artificial setups where the links between features are known. The paper does not test how predictive closure shows up in large models on real video, robotics data or world models. Still, the framework is useful because it provides a language for a question that was previously answered by intuition.

What to check in practice

Teams that train JEPA-style models for video, robotics and planning get a concrete set of hypotheses. First, when diagnosing lost features, look at their role in predicting other features, not only at their own predictability. Second, if an important feature is poorly encoded, try strengthening a feature linked to it with an auxiliary objective or side information. Third, when changing the composition of the input data or the prediction conditions, check whether other features that depended on the removed links have suffered.

For the industry, this is a step from heuristics toward a theory of feature selection in one of the most discussed self-supervised learning architectures. The repository with the experiments is open, so the results can be reproduced and adapted to your own tasks.