SWIM saves compute: audio-video generation gains up to 30.7% over Best-of-N

A team of researchers has published SWIM (Spend Where It Matters) on arXiv, a method that improves joint audio-video generation without fine-tuning the model and without increasing the compute budget. On the JavisBench-Mini benchmark, at comparable cost, the average relative gain across seven metrics for JavisDiT++ was 30.7% over Best-of-N and 14.1% over EvoSearch. For LTX-2 the numbers differ: 9.2% over Best-of-N and 17.0% over EvoSearch.

Why inference-time alignment is expensive

Reward-guided inference-time alignment is an approach in which the model is not retrained. Instead, generation searches for the output that maximizes a given reward. Joint audio-video generation usually involves several rewards: image quality, sound quality, how well the two are synchronized, and so on. Scoring each candidate requires a run of a heavy model, so under limited resources conventional schemes quickly hit the budget ceiling.

Existing approaches have opposite weaknesses. Particle-based search explores the space well globally, but a finite population introduces estimation error. The Metropolis-Hastings (MH) algorithm can improve a candidate locally, but it needs a good starting point. The SWIM authors combined both approaches in a single two-stage scheme.

How SWIM works

The framework requires no training and does not depend on a specific reward function. In the first stage, particles are allocated adaptively across checkpoints of the denoising process, based on online variance estimates. Compute concentrates where it is most useful, and the variance of the estimate is minimized.

The first stage yields a promising candidate. In the second stage, MH works with the finished outputs: proposals are built through local noising and repeated denoising, and steps are accepted based on the reward. Because the starting point is strong, MH extracts additional reward without increasing the number of particles and without an expensive global search. According to the authors, at equal compute the combination of search and MH refinement outperforms either approach alone.

What the experiments showed

• JavisDiT++ on JavisBench-Mini: +30.7% over Best-of-N and +14.1% over EvoSearch (average relative improvement across seven metrics).
• LTX-2 on JavisBench-Mini: +9.2% over Best-of-N and +17.0% over EvoSearch.
• SD1.5, image generation: adaptive particle allocation gave +2.2% on AestheticScore and +6.0% on ImageReward compared with uniform allocation.
• The comparison was run at closely matched compute costs.

One detail stands out: on LTX-2, SWIM's lead over Best-of-N is smaller than on JavisDiT++, but its lead over the evolutionary EvoSearch is larger. The gain depends on the base model, and there is no universal improvement factor. The SD1.5 results cover only the first stage, that is, adaptive allocation. MH refinement was not evaluated in that experiment.

Where this is useful

For teams building pipelines for video generation with sound, the main takeaway is practical: given a fixed limit of GPU-hours, it pays off more to allocate particles across denoising steps by variance than to allocate them uniformly or to rely entirely on Best-of-N. If the reward is not differentiable or the model is closed, the method still applies, since it does not access gradients and does not require fine-tuning.

The limitations are also clear. The evaluation used JavisBench-Mini, a reduced version of the benchmark, and the set of models is limited to JavisDiT++, LTX-2 and SD1.5. Before moving the approach into production, it is worth testing it on your own rewards and the full range of tasks. But the underlying principle, spending the budget where it actually reduces error, is general and transfers easily to other diffusion pipelines.