MultiMDM: A Diffusion Language Model That Generates Text in a Few Steps

Researchers have introduced MultiMDM (multi-mask diffusion model), a diffusion language model that uses several masks instead of one, which lets it generate text in a small number of steps without losing quality. The preprint was published on arXiv under cs.CL in July 2026.

Why few-step generation stalls

Standard masked diffusion models (MDM) perform poorly in few-step mode because all of their forward trajectories collapse into a single, fully masked state. At that point there is no "residual entropy" left for consistency-based approaches to rely on when generating in a small number of steps, so the model has nothing to hold on to.

The alternative, uniform-state diffusion, avoids this degeneration but pays a different price: it is harder to tell clean tokens from noise. According to arXiv, this typically hurts both generation quality and training efficiency. MultiMDM addresses both problems at once while staying within the masked paradigm.

Speed here is not cosmetic: the fewer the steps, the cheaper and faster the inference, which is why few-step mode is key to making diffusion LLMs practical.

What MultiMDM proposes

MultiMDM keeps the masking structure but introduces a set of several masks instead of a single one. In the forward process, each clean token is first pushed toward its assigned mask and then gradually mixed across the whole set of masks.

As a result, the backward process gains the ability to "draft": the model first predicts the assigned mask and only then refines it into a clean token. This two-stage movement is what allows it to get by with few steps.

Unlike the single mask of classic MDMs, the set of masks leaves the model "room to maneuver": the very residual entropy that few-step mode was missing.

• The model is MultiMDM (multi-mask diffusion model), an arXiv preprint (cs.CL) from July 2026
• The key idea is several masks instead of one, which preserves the masking structure for few-step generation
• The backward process works in two beats: first a draft (mask prediction), then refinement into a clean token
• Training uses a closed-form ELBO that allows continual training on top of already pretrained MDMs
• Distillation is purely discrete-state consistency distillation with shared-Gumbel coupling

How it is trained

The authors derived a closed-form ELBO, a training objective in closed form that makes it possible to fine-tune MultiMDM on top of existing pretrained MDMs rather than train it from scratch. This noticeably lowers the cost of entry: existing models become the starting point.

The second component is purely discrete-state consistency distillation with shared-Gumbel coupling, which reduces pathwise entropy. Distillation is what "compresses" multi-step generation into a few steps while preserving quality.

"MultiMDM provides an efficient foundation for principled few-step

generation," the paper's abstract on arXiv says.

What this means

Diffusion language models are the main alternative to conventional autoregressive models, which write text strictly token by token. Diffusion can generate an entire passage at once and refine it over several passes, and MultiMDM is a step toward doing that quickly and without a drop in quality, building on models that are already trained.

Frequently asked questions

How does

MultiMDM differ from standard masked diffusion models?

MultiMDM uses not one mask but a whole set. This preserves the masking structure and gives the backward process the ability to draft (first predict the mask, then refine the token), which classic MDMs, with their single fully masked state, lack.

Does MultiMDM have to be trained from scratch?

No. According to arXiv, the closed-form ELBO allows continual training on top of already pretrained MDMs, meaning existing models serve as a starting point instead of a new model being trained from a blank slate.