LensVLM: A Neural Network for Reading Text in Compressed Images

Apple researchers have developed LensVLM, a neural network that learns to reliably read text in heavily compressed images, addressing the problem of processing long documents through the visual channel.

Why text as an image is promising

Modern vision-language models (VLMs) can take in text not through tokenization but directly, as a rendered image: they literally "see" what is written. This differs fundamentally from how traditional language models work.

In the classic approach, each character or subword becomes a numerical token, and a long document produces a long sequence. That strains memory and limits the context window. VLM visual encoders work differently: they always produce a fixed number of tokens from an image, regardless of how much text it contains.

This suggests a tempting idea: by lowering the rendering resolution, you can fit more text into the same "token budget." Resolution becomes a flexible dial for the degree of compression. For books, PDF reports and other long documents, this approach is potentially far more economical than standard tokenization.

Why does high compression break accuracy?

Under aggressive compression, VLM accuracy drops sharply. The cause is mechanical: characters in the image shrink below the effective resolution of the visual encoder. Letters blur together and become indistinguishable, and the model loses the ability to read the text, even though it technically "sees" the image.

The result is an unwelcome trade-off: the higher the compression, the more text fits into the context, but the worse the model understands it. This barrier has so far limited the practical use of the visual approach for long documents.

How does LensVLM solve the problem?

Apple ML Research's key idea is a mechanism called selective context expansion. Instead of processing the whole image uniformly, the model learns to scan it adaptively, giving extra attention to regions with dense or small text.

The name LensVLM alludes to the optical lens: just as a lens focuses on a particular detail and ignores the rest, the model "sharpens focus" on critical fragments. As a result, overall compression stays high, while local resolution increases only where it is truly needed.

The framework has two complementary components:

• Inference framework — defines how the model examines the image at inference time: when and by how much to increase attention to a specific region
• Post-training recipe — adapts existing VLMs to work with compressed text representations, without training from scratch

This makes LensVLM practical: it can be built into existing models without developing a new architecture from scratch.

What this means

LensVLM shows that the visual channel can become a full alternative to tokenization for long documents. If the approach proves scalable, it may change how VLMs handle books, PDF files and other documents: fewer tokens, a wider context, higher reading accuracy.

Can a neural network read text in compressed images?

Yes, LensVLM is trained specifically to do this. It perceives text as a rendered image and can reliably recognize it even under heavy compression.