AIRI раскрыла слабость token cramming: 1568 токенов в векторе, но понимание теряется

Researchers from the FusionBrain AIRI laboratory presented at ICML 2026 the paper "Progressive Cramming: Reliable Token Compression and What It Reveals" and proved that perfect word-by-word text reconstruction from a compressed embedding does not mean the language model retains the ability to reason over it.

Where token cramming came from

In February 2025, AIRI researcher Yuri Kuratov and his team published the paper "Cramming 1568 Tokens into a Single Vector and Back Again" and demonstrated something almost impossible: up to 1568 tokens — several paragraphs of text — can be "packed" into a single input embedding of a frozen LLM and then recovered word for word. A vector of several thousand numbers replaced thousands of tokens, opening fantastic prospects for context compression.

Key parameters of the original method:

• 1568 tokens fit into a single input vector
• The model recovers the text with zero token loss
• The frozen LLM is not fine-tuned — only the input embedding changes
• The method was named token cramming

Why reconstruction is not understanding

The FusionBrain AIRI team, while reproducing the result, discovered a hidden problem. The standard token cramming protocol only checks the accuracy of text reconstruction, not how the model understands and uses the content.

The authors of the new study proposed an alternative protocol — progressive cramming: compression starts with a small number of tokens and gradually increases, while at each step not only the reconstruction accuracy is checked, but also the model's ability to perform comprehension tasks — question-answer pairs, logical chains.

"The standard token cramming protocol conceals catastrophic failures.

It is one thing to reconstruct text, and quite another to understand it," — Dmitry Tarasov, FusionBrain AIRI researcher, in the laboratory's publication on Habr.

The results were discouraging: at high compression rates, the model copes with verbatim reproduction of the text, but completely loses the ability to reason about its content. Accuracy on comprehension tasks drops to chance level.

Where the failure hides

AIRI found that the failure occurs not in the deep layers of the transformer, but in the first layers of the network — it is these layers that are responsible for the primary processing of tokens and the formation of semantic features.

With token cramming, the optimizer learns to bypass the first layers: a vector is created that allows subsequent layers to "guess" the required tokens without understanding their meaning. In effect, the model learns to compress, bypassing semantics. According to the study's authors, analysis of optimization trajectories showed that the embedding for 1568 tokens lies in a completely different region of the space than normal input vectors — this explains the model's inability to "think" with such data.

What this means

Token cramming in its current form is an impressive reconstruction trick, but not a working method of context compression for tasks requiring understanding. The FusionBrain AIRI work changes the standard for evaluating compression methods: it is not enough to check reconstruction accuracy — it is necessary to check that the model's ability to reason is preserved.

Frequently asked questions

What is token cramming?

Token cramming is a method in which hundreds or thousands of text tokens are "packed" into a single input vector of a frozen language model. The original work by Yuri Kuratov (AIRI, February 2025) demonstrated the possibility of recovering 1568 tokens from a single vector without fine-tuning the model.

How does progressive cramming differ from the standard protocol?

Progressive cramming is an evaluation protocol proposed by AIRI at ICML 2026, in which the compression ratio increases gradually and at each step not only reconstruction accuracy is checked, but also the model's ability to solve comprehension tasks. It was precisely this protocol that revealed that perfect reconstruction does not mean reasoning is preserved.