Text autoencoder compresses web text to 2.24 bits per byte without losing meaning

Researchers have posted arXiv preprint 2610.10738 describing a text autoencoder that compresses text into a short discrete latent representation. On web text, the model reaches 2.24 bits per byte. Raw text takes 8 bits per byte, so that is roughly 3.6x compression. According to the authors, the result is comparable to lossless compression algorithms. The reconstructed text stays close to the original, and the representations are usable for downstream tasks.

How the architecture works

The work sits at the intersection of classical data compression and representation learning. The encoder shortens the sequence of hidden states along the time axis, so N tokens become noticeably fewer vectors. The decoder then stretches the sequence back to its original length. Both operations, downscaling and upscaling, use residual connections so that information is not lost in intermediate layers.

The key component is in the middle: a low-dimensional residual discrete bottleneck, where the compressed vectors are quantized into a finite set of codes. The discreteness matters. Only such codes yield a bit string and allow an honest measurement of how many bits are spent per byte of source text.

The authors did not stop at a single configuration. They compare different quantization methods, different training objectives and different datasets. The abstract does not say which variants were included in the comparison; that will have to be checked in the paper itself.

How quality was measured

Lossy compression needs two yardsticks. The first shows how closely the reconstructed text matches the original on the surface; the authors used BLEU for this. The second shows whether the meaning survived, and an LLM judge plays that role. If the model rephrases a sentence, BLEU will drop, while the judge may still count it as correct.

At several compression levels, the authors compare both metrics. They also run the models through question-answering and semantic text similarity benchmarks. This checks that the latent works as a usable representation, and not only as an archive from which the text can be restored.

Key parameters from the abstract:

• Task: lossy text compression via an autoencoder with a discrete latent
• Mechanism: residual shortening and lengthening along the time axis
• Bottleneck: residual low-dimensional discrete bottleneck
• Result: 2.24 bits per byte on web text
• Benchmark: the level of lossless compression algorithms
• Metrics: BLEU, LLM judge, QA, semantic text similarity

What is still unknown and where this could be useful

This is the first version of the preprint, and it has not been peer reviewed. The abstract gives no BLEU values, no judge scores, no model sizes and no encoding or decoding speeds. It is also unclear which lossless compressors the comparison used. The phrase "on par with" says nothing about the quality margin until the tables are visible.

There is still practical interest. If the latent really preserves meaning under heavy length compression, it has several obvious uses:

• Context compression for LLMs. A short sequence of latent vectors in place of long text could make working with long documents cheaper.
• Corpus storage. For search and retrieval, meaning matters more than letter-by-letter matching, so losses are acceptable here.
• Semantic embeddings. The semantic text similarity test hints that the representations can be used directly.

These are hypotheses, not the authors' conclusions. Three things are worth checking: how reconstruction error grows under more aggressive compression, how robust the method is on text outside the web domain (code, tables, rare languages), and what encoding and decoding cost. Until the full paper has been read, the result should be considered promising but not confirmed.