Distilled Diffusion Models Learn to Forget Unwanted Concepts via DPO

Fast text-to-image models, compressed to a few generation steps instead of dozens, have become the standard for low-latency products. Along with their speed, they inherited the base models' ability to draw things they should not: specific people, nudity, protected objects. The authors of a new arXiv preprint (cs.CV) claim that existing methods for removing such concepts work poorly on distilled models, and they propose their own approach based on Direct Preference Optimization.

Why classic unlearning breaks on distilled models

Most data-driven unlearning methods fine-tune the model's weights with specialized objective functions. These functions implicitly rely on the dynamics of multi-step denoising: the model cleans up the image gradually, and the "forget" signal is spread along a long trajectory. In few-step distilled (FSD) models, the trajectory is compressed to a handful of steps, the generation dynamics differ, and that assumption no longer holds. The result, according to the authors, is ineffective forgetting: the target concept still shows up.

The obvious workaround is to run unlearning on the undistilled base model and then distill again. But this takes significant compute and time, and in many scenarios the process is impractical, especially when the base model is updated and the list of banned concepts keeps growing.

What the authors propose

The researchers returned to Direct Preference Optimization for diffusion models. Standard DPO and its unlearning derivatives are formulated around noise-prediction error. On FSD models, that formulation transfers poorly precisely because of the changed generation dynamics.

Instead, the authors introduce a modified preference optimization formulation that is explicitly aligned with the properties of few-step generation. The idea is to remove the concept directly in the distilled model, without touching its speed and without destroying its other skills.

Parameters of the work disclosed in the abstract:

• Subject: few-step distilled text-to-image diffusion models.
• Method: preference-driven unlearning based on DPO reformulated for FSD generation.
• Main tasks: removing identities and NSFW content (nudity).
• Additionally: object-level unlearning.
• Claimed properties: robust forgetting of target concepts, preservation of the few-step regime, strong retention of "allowed" capabilities.
• The abstract gives no numerical metrics; they have to be checked in the full text of the paper.

What teams shipping fast models should do

For those deploying distilled models in production, the main takeaway is practical: transferring safety methods from the base model to the distilled one cannot be assumed to work automatically. Testing has to happen on the FSD model itself and at the step counts actually used in the service. If a filter made the model "forget" a concept at 50 steps, that guarantees nothing for 1–4.

For now, the work exists as a preprint with no independent verification. Before adopting it, it is worth waiting for the code and checkpoints, running your own red-team prompt sets, and measuring side-effect degradation: for example, how removing a specific identity affects generation of people who look similar. It also makes sense to compare the approach's cost with re-distillation on your own hardware, since the savings in time and compute are the main argument here.

For regulatory and legal scenarios, from requests to remove biometric data and a person's likeness to NSFW moderation, the ability to edit an already compressed model directly looks considerably more convenient than rebuilding the entire pipeline. The authors do not say in the abstract how robust the method is against workaround prompts or against fine-tuning by an attacker, and that question remains open.