KAIST Teaches Diffusion VLMs to Stop Looking at the Image: 35% of Compute at 99.6% of Quality

A group at KAIST has proposed a way to speed up diffusion multimodal models. These are neural networks that understand images and text and generate answers not word by word but in batches, gradually "unmasking" masked tokens. The method is called LOWN (Look Only When Needed). Across ten benchmarks, it retains 99.6% of LLaDA-V's scores at 35.3% of the model's decoding compute. It requires neither fine-tuning the model nor tuning thresholds.

Why the model keeps recomputing the same image

Diffusion VLMs (DVLMs) have a structural quirk. Their attention is bidirectional, so the exact key-value cache familiar from autoregressive models does not work on them directly. Standard inference runs the entire sequence at every step: image, prompt and answer. Visual tokens make up 89–97% of that sequence, while a pass without them costs only 3–10% of a full one. Most of the compute, in other words, goes to reprocessing an image that never changes.

The authors checked how often the image is actually needed. They compared LLaDA-V's predictions with and without the image along the same decoding trajectory, on all ten benchmarks:

• For 83.4% of the positions the model was about to unmask, the most probable token was the same in both modes. • 85.0% of final tokens had already appeared as the top prediction at their position before the moment they were unmasked. • The image changed the unmasked token on only 22.8% of steps on average. The spread is wide: 7.9% on MMBench and 43.8% on DocVQA, where the model has to read text from scans.

On eight of the ten benchmarks, the divergences are concentrated at the start of generation, but the pattern is not universal; DeCapBench is one exception. That is why a fixed schedule of "look at the image every N-th step" works poorly: the need for the image depends on both the task and the specific step.

How LOWN decides whether the image is needed

After each full pass (with the image), the method saves its predictions and confidence as an "anchor." On the following steps, it runs a cheap pass without visual tokens, keeping the position indices intact. The procedure then works like this:

• Masked positions are sorted by the anchor's confidence. • A token counts as confirmed if the prediction without the image matches the anchor. • The method takes the longest prefix of consecutive confirmed tokens. The first mismatch cuts it off. • If the prefix is at least as long as the usual number of tokens per step, the full pass is skipped. The whole prefix is accepted, so a single step can unmask more tokens than the base model would. • If the prefix is shorter, a full pass runs. It updates the anchor, and the positions where the predictions with and without the image diverged are unmasked first.

The method never takes more steps than the base model, and the length of the accepted prefix is determined by the input itself, with no manually set threshold. Confirmation is sometimes wrong: a fresh full pass reverts 2.2–4.7% of the tokens unmasked on steps without the image. To keep errors from accumulating, the anchor is replaced entirely at every full pass.

What the experiments showed

The evaluation covered ten benchmarks: multiple-choice tests, document and chart understanding, math problems and detailed image descriptions. Among acceleration methods with a budget at least as large as LOWN's, it leads on eight of the ten. The comparison included visual token pruning (RedVTP, ZOO-Prune), the KLASS unmasking schedule and the Fast-dLLM cache.

An important detail: LOWN does not replace other methods but stacks with them. It changes how often a full pass is needed, while pruning and caching make the pass itself cheaper. Combined with pruning or caching, it cuts the partner method's cost by a factor of 1.8–2.5. Paired with a cache, decoding is 5.6–7.0 times faster than with the original model. The same rule carries over to LaViDa and VidLaDA, that is, to other DVLMs and to video.

What this means for engineers

If you run diffusion VLMs on documents or charts, the main cost item is repeated image processing. LOWN reduces it without retraining or per-task hyperparameter tuning, so it can be plugged in at the inference stage. One limitation is worth keeping in mind: the gain depends on the benchmark. The more visual tokens in the request and the less often the answer needs the image, the bigger the savings. On tasks like DocVQA, where the image is needed on nearly half of the steps, the savings are more modest. The authors have released the code and a project page.