A Small JEPA Model Matches Next-Token Prediction With 21.6M Examples Instead of 900M

Language model pretraining comes down to one constraint: how many examples you have to run through the model. The standard recipe, forcing it to predict the next token (essentially, the next word), requires hundreds of millions of texts. Randall Balestriero and Yann LeCun show there is another way.

Their 109M-parameter text model, trained with JEPA on "good views," matched next-token prediction on alignment with images: 10.9% vs 11.1%.

It did so after seeing 21.6M examples instead of 900M, which is 42 times fewer. Imagine you are training search over a product catalog.

Instead of feeding the model gigabytes of random descriptions, you give it pairs of "product photo and its text." Those are exactly the kind of pairs this paper is about.

What makes a "good" view and a "bad" one

JEPA is a family of self-supervised models. They are trained so that the representations (embeddings) of two "positive" examples land close together, without collapsing to a single point. What matters most is where the positive pairs come from.

Take a photo of a cat on a sofa. A "bad view" is the same photo cropped, blurred or with altered brightness. These are distortions of the same observation: the model easily learns to ignore them, but learns little about the world. A "good view" is a different observation of the same scene: a shot from another angle, or a text description of that very picture. To link such examples, the model has to pick out what actually underlies the scene.

The authors prove a theorem about "Platonic representations" for JEPA. The Platonic hypothesis holds that different models trained on different data converge on a similar internal representation of the world. The theorem explains the conditions under which this happens for JEPA. The authors also show that next-token prediction is mathematically a JEPA with bad views: the context and the continuation are two pieces of the same text.

The results in numbers

The comparison used the cross-modal benchmark WIT-1024, paired with the DINOv2 ViT-B image encoder. The metric is mutual 10-NN: for the same objects, it counts how many of the ten nearest neighbors coincide in the text space and the image space. The higher the share, the better the two modalities are aligned.

• The 109M-parameter text JEPA with good views scored 10.9% on 21.6M examples. Next-token prediction on the same corpus scored 11.1% but needed 900M examples.
• When the model was trained on texts for 940M web images, it reached 18.4%. Qwen2.5-72B scored 15.4%, Llama-3-70B 15.1%, and the CLIP text encoder 12.4%.

There are also side results. Two separate unimodal encoders, one for text and one for images, can be connected with a single orthonormal projection. This pairing writes image captions with a CIDEr of 0.838 on COCO. The modalities can also be aligned with no paired data at all: top-5 accuracy on ImageNet-100 was 64.4%, against 47.6% for Qwen3-8B. The code, the proof and a demo are published.

What to keep in mind and what to do

First, the metric measures how well text aligns with images, not the overall quality of a language model. You cannot read the result as "a 109M model is smarter than Llama-3-70B": it does not test the ability to write code or reason.

Second, and most important, the limitation: the gain on web texts relies on views grouped by image ID and on captions generated by a pretrained image-captioning model. Another model has already done part of the work, and its cost is not reflected in the "42 times fewer examples" comparison. For tasks with no natural "scene" containing several observations, the recipe does not transfer directly. This is a preprint, and there are no independent reproductions yet.

The practical takeaway is for those building multimodal search or recommendations, or training encoders for their own domain. Instead of growing the corpus, think about how to assemble pairs from different descriptions of the same object: a product photo and its text, an interface screenshot and a scenario description, a documentation fragment and a question about it. If you already have such groups, a small model may deliver quality that you would normally go to models an order of magnitude larger to get.

For everyone else, this is a signal that the debate of "scale versus data structure" is not settled. If the result holds up on other benchmarks and modalities, next-token prediction will no longer be the only reasonable pretraining recipe.