MultiMDM: диффузионная языковая модель для генерации текста за несколько шагов

Researchers have presented MultiMDM (multi-mask diffusion model) — a diffusion language model that uses multiple masks instead of one, and thanks to this generates text in a small number of steps without loss of quality. The preprint was published on arXiv in the cs.CL section in July 2026.

Why few-step generation stalls

Ordinary masked diffusion models (MDM) perform poorly in the few-step regime, because all their forward trajectories collapse into a single fully masked state. At that point there is no "residual entropy" left for consistency-based approaches to rely on for generation in a small number of steps — the model simply has nothing to grab onto.

The alternative in the form of uniform-state diffusion avoids this degeneration, but pays a different price: there it is harder to distinguish clean tokens from noise. According to arXiv, this usually hurts both generation quality and training efficiency. MultiMDM solves both problems at once, while staying inside the "masked" paradigm.

Speed here is not cosmetic: the fewer the steps, the cheaper and faster the inference, so the few-step regime is the key to the practicality of diffusion LLMs.

What MultiMDM proposes

MultiMDM preserves the masking structure, but introduces a set of several masks instead of a single one. In the forward process, each clean token is first pushed toward its assigned mask, and then gradually mixed across the whole set of masks.

Because of this, the backward process gains the ability to "draft": the model first predicts the assigned mask, and only then refines it down to the clean token. It is precisely this two-stage movement that allows it to get by with a small number of steps.

Unlike the single mask of classic MDMs, the set of masks leaves the model "room to maneuver" — the very residual entropy that was missing for the few-step regime.

• Model — MultiMDM (multi-mask diffusion model), preprint on arXiv (cs.CL), July 2026
• Key idea — several masks instead of one, to preserve the masking structure for few-step generation
• The backward process works in two beats: first a draft (mask prediction), then refinement down to the clean token
• Training — closed-form ELBO, allowing continual training on top of already pretrained MDMs
• Distillation — purely discrete-state consistency distillation with shared-Gumbel coupling

How it is trained

The authors derived a closed-form ELBO — a training objective in closed form that makes it possible not to train MultiMDM from scratch, but to fine-tune it on top of already ready pretrained MDMs. This noticeably lowers the barrier to entry: existing models become the starting point.

The second component is purely discrete-state consistency distillation with shared-Gumbel coupling, which reduces pathwise entropy. Distillation is needed precisely in order to "compress" multi-step generation into a few steps while preserving quality.

"MultiMDM provides an efficient foundation for principled few-step

generation," the paper's abstract on arXiv states.

What this means

Diffusion language models are the main alternative to the familiar autoregressive models, which write text strictly token by token. Diffusion is able to generate an entire fragment at once and refine it over several passes, and MultiMDM is a step toward doing this quickly and without a drop in quality, building on models that are already trained.

Frequently asked questions

How does

MultiMDM differ from ordinary masked diffusion models?

MultiMDM uses not one mask but an entire set. This preserves the "masked" structure and gives the backward process the ability to draft — first predict the mask, then refine the token — something classic MDMs lack with their single fully masked state.

Does MultiMDM need to be trained from scratch?

No. According to arXiv, closed-form ELBO allows continual training on top of already pretrained MDMs, meaning existing models can be used as a starting point instead of training a new model from a blank slate.