Consistent Distribution Matching distills diffusion to 1 step with FID 2.04

Diffusion and flow models generate high-quality images, but they pay for it with the number of numerical integration steps. A group of authors has proposed Consistent Distribution Matching (CDM), a distillation method that speeds up such models without a dataset, without teacher runs, and without auxiliary networks. On ImageNet 256×256, the student reaches FID 2.04 with one function evaluation (1-NFE) and FID 1.37 with four (4-NFE). Training takes 40 epochs. According to the paper, the result outperforms the best data-free distillation baselines. Code and model are published at consistentdmd.github.io.

What is wrong with current distillation

The idea of distillation is simple: the student learns to reproduce the teacher's dynamics so that it can produce in one or two steps what the teacher needs dozens for. In practice, existing approaches come with a long tail of dependencies. Some need a specially assembled distillation dataset. Others need expensive teacher runs, meaning full trajectory simulation, to produce training pairs. Still others require auxiliary proxy networks that have to be trained alongside the main model.

Each of these dependencies makes scaling harder. The dataset has to be stored and cleaned, teacher runs eat compute, and an extra network adds hyperparameters and the risk of unstable training. The CDM authors target exactly this layer of engineering overhead.

How CDM works

The key move: a single student network does two jobs at once, generating samples and estimating the score (the gradient of the log density). In schemes with a separate proxy network, these functions are usually split apart. Here they are merged, so only two models take part in training: a frozen teacher and a trainable student. A single objective is optimized.

The paper describes the student as a flow-map, that is, a mapping that carries noise to a sample. The authors prove that minimizing their objective implies Wasserstein convergence: the distributions obtained by pushing forward (pushforward) through the student's flow-map approach the teacher's marginal distributions. This is a theoretical guarantee that the student fits the teacher's distribution, not just metric-friendly points.

The method is called data-free and simulation-free. Training requires neither real images nor running the teacher along the full trajectory.

Key metrics

• Benchmark: ImageNet 256×256.
• 1-NFE: FID 2.04.
• 4-NFE: FID 1.37.
• Training budget: 40 epochs.
• Components: a frozen teacher and a trainable student, one loss function.
• Distillation data: not used.
• Theory: Wasserstein convergence is proved for the flow-map pushforward.
• Release: code and model are openly available.

What to check in practice

All the numbers in the paper come from class-conditional generation on ImageNet 256×256. That is a standard but fairly narrow testbed. There is no basis for extending the result to text-to-image models with billions of parameters, long prompts, and high resolution. Whether CDM scales will be the main question about it. The authors' claim of beating the best data-free baselines should be checked against the tables in the paper itself and against reproduction with the open code.

For teams building production inference with image or video generation, the method is interesting for three reasons. First, there is no need to prepare a dataset for distillation, which removes a separate problem for closed teachers or ones with tricky licensing. Second, a pipeline with two models and one loss is easier to maintain and cheaper to debug. Third, one step at FID 2.04 and four steps at FID 1.37 point to a convenient trade-off: you can choose between maximum speed and quality without retraining from scratch.

A logical first step for a team: take the published code, repeat the distillation on a small flow model of your own, and measure FID at 1 and 4 steps. It is worth comparing GPU-hour costs against your current recipe right away, because savings on data and teacher runs matter as much as the metric itself.