A 109M-parameter JEPA beats Qwen2.5-72B: alignment of 0.184 versus 0.154

The Platonic Representation Hypothesis (PRH) holds that as models grow, their internal representations of the world, built from text and from images, become increasingly similar. Until now this was only an observation: for the self-supervised tasks used in practice, no rigorous proof existed. A new paper turns PRH into a theorem for the JEPA family (Joint-Embedding Predictive Architectures, in which the embedding of one kind of data predicts the embedding of another).

The practical result: a 109M-parameter text JEPA scored 0.184 on the alignment metric with vision models on the benchmark from the PRH paper. That is higher than any language model tested, including Qwen2.

5-72B at 0.154, and higher than the CLIP text encoder (0.124).

The lead held across two benchmarks and two metrics.

A theorem on when modalities coincide

The proof idea is short. A JEPA with a "whitened" second-order anti-collapse term learns the spectral embedding of the graph defined by its positive pairs, that is, pairs of inputs declared to be views of each other. If different modalities (text, images, video, audio, actions) describe the same world, and their positive pairs link the same world states, then the training problems coincide up to a change of coordinates. The solutions also coincide up to a single rotation. This is the Platonic Representation Theorem, and it holds for any number of modalities.

If the modalities' positives differ, the misalignment is bounded by the divergence between their positive-pair graphs and by the size of the spectral gap. The authors tested the bound on 300 randomly generated finite worlds: in all of them the error fell below the bound, at no more than 18% of it.

The practical takeaway: it is not only model and data size that matter, but also what counts as a positive pair. A "good view" is two observations of the same scene, for example two descriptions of one photo. A "bad view" is two corruptions of the same observation: two crops of a photo or two maskings of one sentence. In the second case the encoder spends some dimensions on quirks of its own modality, and no rotation will align it with another modality.

By this logic, next-token prediction is a JEPA with a bad view: the positive is a prefix and the continuation of the same document. The authors argue this is why language models converge with vision only partially, and mostly through scale.

What the experiments showed

The tests used identical texts and encoders. Good views gave better alignment in almost every comparison: from 145K Flickr captions to the texts of 940M web images, with image encoders both trained from scratch and frozen, and on video as well. Alignment follows a scaling law in the share of good views and the number of examples seen. It predicted 85 new runs before they were launched as accurately as a rerun with a different seed.

Key numbers on the benchmark from the PRH paper:

• a 33M-parameter text JEPA trained on the texts of 50M web images scored 0.125 versus 0.034 on Flickr captions, on par with Qwen2.5-3B and Qwen2.5-7B;
• on the same corpus, a JEPA with good views reached with 21.6M training examples what next-token prediction reaches with 900M, making it 42 times more efficient;
• a 109M-parameter JEPA that read 116B tokens beat models 660 times larger trained on more than 100 times as much data;
• paired with the CLIP ViT-H image encoder, it scored 0.230 versus 0.210 for the best language models; the lead holds with MAE and I-JEPA encoders as well.

Captions without pairs, and the limits of the theory

Alignment also works as a tool. A single linear map with orthonormal rows, the only part trained on pairs, connects frozen DINOv2 and DINOv3 to a language model fine-tuned on text alone. The result: a CIDEr score (an image captioning quality metric) of 0.838 on COCO. For DINOv2 ViT-B this map has 0.22M parameters; for the strongest captioner with 5120-dimensional features, 1.9M.

With no pairs at all, the rotation can be found from the geometry of unpaired images and captions. On a benchmark for this kind of alignment, where results are judged by zero-shot classification, the 109M-parameter text JEPA surpassed all published accuracies on 21 pairs of vision and language models across five settings. It beat the strongest language model, Qwen3-8B, rerun under the same conditions, by 17 points of zero-shot accuracy on ImageNet-100.

The authors state the limits of the theory themselves. The theorem is proven for an idealized model: a finite world, injective rendering, population optima, an identity predictor. Uniqueness of the solution under a constraint on the whole distribution, as in SIGReg from LeJEPA, is taken as an assumption. The theory does not cover JEPA world models with an action-conditioned predictor, nor the case of a linear map between embeddings of different dimensions.

For practitioners, the main takeaway is that when collecting data for multimodal systems, it is worth thinking about pairs of "several independent observations of one scene," and not only about corpus size.