TTR-Compressor Compresses Sequences with Regression and Cuts Visual Tokens by 55.6%

Researchers have presented TTR-Compressor, a sequence compression module that uses regression solved directly in the forward pass instead of pooling and learned projections. In multimodal language models, it consistently outperforms the standard MLP-merger and matches its quality with 55.6% fewer visual tokens. In block-sparse attention, it substantially improves the RULER benchmark result compared with compressors built on the attention mechanism itself.

A bottleneck inside efficient models

Sequence compression shows up everywhere in modern architectures, even though it rarely gets much attention. Visual encoders produce hundreds of tokens per image, and these must be merged into a smaller number before they go into the language model. In sparse attention, a long context is split into blocks, and each block needs a compact representation that the model uses to decide where to look. The same tricks are used everywhere: averaging, max-pooling, or a small learned projection such as an MLP.

The problem is that these operations lose information exactly where it matters most. Averaging blurs rare but important tokens. A fixed projection cannot adapt to the content of a specific sequence. The authors explicitly call compression a critical information bottleneck and describe common approaches as not strong enough.

Regression solved on the fly

The idea behind TTR-Compressor is to treat compression as a regression problem at test time. The input tokens themselves define the regression task, and the fitted coefficients serve as the compressed representation of the block. Information about the sequence reaches the output by fitting to its own tokens, not through a fixed linear mapping.

The key engineering detail concerns computational cost. The formulation of the regression problem is trained end to end with ordinary backpropagation. The coefficients themselves, however, are computed in closed form in the forward pass, with no inner optimization iterations. This design gives meaningful input adaptivity while keeping the speed that makes a compressor worth using in the first place. Because the closed-form solution is differentiable, everything can be trained together with the rest of the model.

Key results from the paper:

• block-sparse attention: TTR-Compressor noticeably improves RULER scores compared with attention-based compressors;
• visual token merging in multimodal LLMs: the method consistently outperforms the standard MLP-merger;
• efficiency: MLP-merger quality is reached with 55.6% fewer visual tokens;
• generality: the authors position the method as a general replacement for pooling and projections, not as a solution for a single task.

Why the 55.6% figure matters

RULER tests how well a model retains and retrieves information over a long context, which makes it an uncomfortable test for any compression scheme. If a block compressor loses details, sparse attention cannot pick the right blocks, and quality drops. A large margin over attention-based compressors suggests that the regression representation preserves more useful signal.

For multimodal models, the benefit is straightforward to calculate. Visual tokens are expensive in both prefill and the KV-cache, and the number of images and video frames per request keeps growing. Cutting their number almost in half at the same quality means less memory, lower latency, and more images in a single context for the same budget. The compressor also replaces just one component, so the rest of the architecture does not need to be rewritten.

Where it could be useful first

The first candidates to test it are teams that train multimodal models with a visual merger and are constrained by inference cost. Replacing the MLP-merger with TTR-Compressor looks like a local experiment: one module changes, and the comparison metric is clear, either quality at a fixed number of tokens or number of tokens at fixed quality.

The second group is developers of long-context models with block-sparse attention. For them, the block compressor determines how accurately attention routing works, and the RULER result points directly to a gain.

There is only one limitation for now: the evaluation covers two classes of tasks. How the method behaves on other modalities, in audio, or in very large models will be shown by independent reproductions. Still, the principle "tokens define the regression, coefficients are the representation" looks like a basic primitive that is easy to build into existing efficient architectures.