109M-Parameter Model Beats Llama-3 70B at Meaning Alignment: What LeCun Proved

A familiar story: we spend months growing LLMs and paying for data centers, yet the model still struggles to grasp what the phrase "a puppy playing by the sea" has in common with a photo of a beach. A new paper by Randall Balestriero and Yann LeCun offers an uncomfortable answer: the problem isn't scale, but how we form training pairs.

What exactly was proved

In 2024, MIT researchers put forward the Platonic Representation Hypothesis: as models grow, their internal representations of different modalities (text, vision, audio) converge. That was an empirical observation. Balestriero and LeCun turned it into a theorem: if different modalities are trained with JEPA on the same positive pairs, their spaces coincide up to an orthogonal rotation.

JEPA (Joint-Embedding Predictive Architecture) predicts neither pixels nor tokens, but vector representations. Unpredictable noise, such as faces in an audience or the exact placement of chairs, is discarded at the encoder level instead of forcing the network to guess.

Good and bad views

The key idea is what counts as a pair. A "good view" is two independent observations of one scene: two different captions of the same photo, or two frames from the same video. Their noise differs, so the model is forced to encode what they share, that is, the state of the world.

A "bad view" is two distortions of the same observation: two masks of one sentence, two crops of one image. The authors also place next-token prediction here: the context and the continuation sit in the same document and share the author's style and vocabulary. Part of the vector's capacity goes to this noise and is lost for alignment with other modalities.

The numbers

The Good-View Text JEPA text encoder, based on BERT-B (109M parameters), was trained on descriptions of 940 million web images. On the WIT-1024 benchmark, paired with a frozen DINOv2, it achieved Mutual 10-NN of 0.184 versus 0.154 for Qwen2.5-72B and 0.151 for Llama-3-70B. It is 42 times more efficient than a next-token-prediction model in terms of samples, and 14 times in terms of tokens.

Two more results for practitioners. First: a caption generator without paired data. A Qwen2.5-1.5B decoder was trained to reconstruct text from a noisy embedding, and it was connected to the DINOv2 visual encoder with a single rotation matrix (the Procrustes problem). The result: CIDEr 0.838 on COCO. Second: alignment with no pairs at all via optimal transport, 64.4% top-5 on ImageNet-100 versus a previous best of 56%.

The authors also derived a scaling law for alignment quality that predicted 85 new runs with an error on par with initialization noise.

Caveats and the catch

The comparison uses a metric of alignment with the visual space, not general intelligence. Llama-3 70B is far from obsolete: it will keep reasoning, writing code and holding conversations better. The benchmarks are mostly about multimodal consistency, and transfer to robotics remains a claim for now, including in connection with the startup AMI Labs. Moreover, this is an adaptation of the paper, and claims of a "dead end" for LLMs should be read as LeCun's position, not a settled question.

Practical takeaway. If you are building multimodal search, RAG over images, or recommendations, think about the composition of your pairs. Several independent descriptions of one object may deliver more than another billion tokens. Technically valuable are random cropping of fragments (without it, quality drops by roughly 25%), gradient clipping under SIGReg regularization, and the ability to join ready-made encoders with a linear rotation instead of costly joint training. For small teams, this is a chance to get strong embeddings on modest hardware.