Omni-Diffusion-Distill speeds up multimodal dLLMs by up to 21x: 8 steps instead of 128

The authors of an arXiv preprint present Omni-Diffusion-Distill, a two-stage method for distilling unified multimodal diffusion language models. It cuts image generation from 128 to 8 decoding steps and multimodal understanding from 512 to 64. In wall-clock time, that is a speedup of 18.2x and 21.2x. The student scores 0.828 on GenEval and 83.0 on DPG-Bench in text-to-image, and 20.0 on MM-Vet (GPT-based score) in understanding tasks.

Why unified dLLMs are slow

A unified multimodal dLLM replaces a pipeline of separate models for image generation and image understanding with a single architecture. Everything runs in a discrete token space, both text and images. The price is iterative decoding. The model gradually refines a mask over tokens, so a single response takes tens or hundreds of forward passes.

Existing few-step distillation methods target only one task: either image generation or text generation. How to compress a fully discrete multimodal dLLM into a single student while keeping both generation and understanding was an open question. This work addresses it.

How the distillation works

In the first stage, the student learns to skip decoding steps. To do this, it reproduces the teacher's cached trajectories. In the second stage, it is fine-tuned on intermediate states taken from its own runs. That way it sees the errors it actually makes under short decoding, not only the teacher's ideal trajectories.

The authors separately address two sources of degradation that arise specifically in the unified setting:

• A pairwise collision penalty reduces repetition in parallel text decoding. When the model picks several tokens in one step, it tends to output the same thing.
• Entropy-matched guidance prevents entropy collapse. It occurs when the student fits the teacher's "sharpened" distribution during image generation.

Numbers from the paper

• Image generation: 128 → 8 steps, 18.2x speedup.
• Multimodal understanding: 512 → 64 steps, 21.2x speedup.
• GenEval: 0.828; DPG-Bench: 83.0.
• MM-Vet (GPT-based score): 20.0.
• COCO captioning: 57.2, twice the teacher's 28.4 at the same step count.

One detail deserves attention. The wall-clock gains (18.2x and 21.2x) are larger than the reduction in step count (16x and 8x). The abstract does not explain why. It may come down to the cost of a single step or to simplified guidance, but that is only a guess, and the answer has to be sought in the full text of the paper. The comparison with the teacher on COCO at an equal step budget also shows that a regular dLLM loses noticeable quality under aggressive step reduction, while the distilled one holds up.

Where this could be useful

For teams building multimodal products on diffusion LLMs, the main takeaway is practical: latency stops being an argument against this class of models. Eight steps per image and 64 for understanding are already closer to interactive use cases than hundreds of passes. The biggest winners are applications where one model has to both draw and describe: assistants, image editors, e-commerce tools.

The caveats are standard for a preprint. The results have not been peer-reviewed and were measured by the authors themselves on selected benchmarks. Absolute MM-Vet scores remain modest. The abstract says nothing about releasing code or weights. Before building the method into production, it is worth waiting for reproduction on other models and testing under real workloads.