DiffusionART speeds up RL fine-tuning of video models 12x, with no EMA tuning

A team of researchers has published DiffusionART on arXiv, a method for online reinforcement learning of diffusion models. It targets two known problems: the slowness of Flow-GRPO and the finicky nature of DiffusionNFT. In the main experiment on HunyuanVideo-13B, where visual quality and motion quality enter the reward with equal weights, DiffusionART reaches the late-stage average reward of DiffusionNFT 12.02 times faster in GPU-hours, and that of Flow-GRPO-Fast 11.72 times faster. Average training reward is also higher, by 28.8% and 53.0%, respectively. This is a preprint, and no independent replications exist yet.

Where earlier approaches broke down

Online RL is needed when a model must be tuned to rewards that cannot be differentiated: aesthetic scores, motion quality, human preferences. Reverse-process methods such as Flow-GRPO work along the denoising trajectory. The reward arrives once for the whole sample, but the trajectory has many steps. It is unclear which step contributed anything, so credit assignment is poor and training is expensive.

Forward-process methods such as DiffusionNFT are more efficient, but they rely on a carefully tuned exponential moving average (EMA) for off-policy training. Without it, training becomes unstable. The DiffusionART authors explain why. By their analysis, NFT-style objectives contain a hidden consistency force between the forward and reverse processes. Joint training on positive and negative examples suppresses it only partially. And EMA stabilizes training not through off-policy rollouts, as is commonly assumed, but through "restoration": it pulls the parameters back when the model drifts.

How DiffusionART works

The name stands for Diffusion Reinforcement with Aggregation-First Reward Tilting. First, the method aggregates the rewards of all samples for a single prompt into a shared improvement direction for the states of the reverse process. This removes the hidden consistency force by construction rather than suppressing it partially. The second step is an additional update with a trust-region mask. It pulls the model back as it drifts, so it takes over the role of EMA without requiring you to tune EMA parameters.

Key details of the work:

• Main test: HunyuanVideo-13B, with visual-quality and motion rewards at equal weights.
• GPU-hour speedup to the late-stage reward level: 12.02× versus DiffusionNFT, 11.72× versus Flow-GRPO-Fast.
• Average training reward: 28.8% higher than DiffusionNFT and 53.0% higher than Flow-GRPO-Fast.
• No differentiable reward model is required.
• The method stays effective with fewer samples per prompt.
• The DiffusionART-Pairwise variant works with pairwise reward models.
• The authors claim it applies to image and video generation without per-task tuning.

What to do with this in practice

For teams that fine-tune video and image models on reward signals, the main value is cost. A run on a 13-billion-parameter video model takes thousands of GPU-hours, so a tenfold reduction changes the experiment budget. Dropping EMA tuning removes one more layer of hyperparameter search, and working with fewer samples per prompt lowers the cost of each step.

The Pairwise variant is useful where there is no scalar score and only "better/worse" comparisons, for example in data from human annotators.

Caution is also warranted. The figures in the abstract describe average training reward and the speed of reaching a given level. They say nothing about how the results look to a human on held-out prompts, and they do not rule out reward hacking, where the model fits the metric rather than actual quality. Before moving the method into your own pipeline, wait for the code, independent reproductions, and evaluations on held-out sets that involve human raters.