AIRI Exposes the Weakness of Token Cramming: 1568 Tokens in One Vector, but Comprehension Is Lost

Researchers at the FusionBrain AIRI lab presented the paper "Progressive Cramming: Reliable Token Compression and What It Reveals" at ICML 2026. Their finding: perfect word-for-word reconstruction of text from a compressed embedding does not mean the language model retains the ability to reason over it.

Where token cramming came from

In February 2025, AIRI researcher Yuri Kuratov and his team published "Cramming 1568 Tokens into a Single Vector and Back Again." The paper showed something that seemed almost impossible: a single input embedding of a frozen LLM can "pack" up to 1568 tokens, several paragraphs of text, and the model can then reconstruct them word for word. A vector with a few thousand dimensions stood in for thousands of tokens, which opened up striking prospects for context compression.

Key parameters of the original method:

• 1568 tokens fit into a single input vector
• The model reconstructs the text with zero token loss
• The frozen LLM is not fine-tuned; only the input embedding changes
• The method was named token cramming

Why reconstruction is not comprehension

When the FusionBrain AIRI team reproduced the result, they found a hidden problem. The standard token cramming protocol checks only how accurately the text is reconstructed, not how well the model understands and uses its content.

The authors of the new study proposed an alternative protocol, progressive cramming. Compression starts with a small number of tokens and increases gradually. At each step, the protocol checks not only reconstruction accuracy but also whether the model can perform comprehension tasks, such as question-answer pairs and logical chains.

"The standard token cramming protocol hides catastrophic failures.

Reconstructing a text is one thing; understanding it is something else entirely," said Dmitry Tarasov, a FusionBrain AIRI researcher, in the lab's post on Habr.

The results were sobering: at high compression ratios, the model handles verbatim reproduction of the text but completely loses the ability to reason about its content. Accuracy on comprehension tasks drops to chance level.

Where the failure hides

AIRI found that the breakdown happens not in the deep layers of the transformer but in the first layers of the network, the ones responsible for the initial processing of tokens and the formation of semantic features.

During token cramming, the optimizer learns to bypass the first layers. It produces a vector that lets the subsequent layers "guess" the right tokens without understanding their meaning. In effect, the model learns to compress while skipping semantics. According to the study's authors, an analysis of optimization trajectories showed that the embedding for 1568 tokens lies in a completely different region of the space than normal input vectors, which explains why the model cannot "think" with such data.

What this means

In its current form, token cramming is an impressive reconstruction trick, but not a working context-compression method for tasks that require understanding. The FusionBrain AIRI paper changes the standard for evaluating compression methods: checking reconstruction accuracy is not enough; you also have to check whether the model's ability to reason is preserved.

Frequently asked questions

What is token cramming?

Token cramming is a method in which hundreds or thousands of text tokens are "packed" into a single input vector of a frozen language model. The original work by Yuri Kuratov (AIRI, February 2025) showed that 1568 tokens can be reconstructed from one vector without fine-tuning the model.

How does progressive cramming differ from the standard protocol?

Progressive cramming is an evaluation protocol proposed by AIRI at ICML 2026 in which the compression ratio increases gradually and each step checks not only reconstruction accuracy but also the model's ability to solve comprehension tasks. This protocol revealed that perfect reconstruction does not mean reasoning is preserved.