UniAVC Unifies Audio and Video Generation and Editing in a Single Set of Weights

UniAVC is a new paper in which one model covers three classes of tasks: joint audio-video synthesis from text, generation of one modality from the other in both directions, and instruction-based editing of both tracks. It all runs on a single set of weights, with no adapters for individual tasks. The authors rely on two design decisions: staged training and separate noise schedules for audio and video. The paper appeared in Hugging Face Daily Papers. The abstract gives no numerical metrics, so it is too early to judge quality by the numbers.

The problem: paired data is scarce

Most existing systems for joint audio-video generation train on paired corpora, meaning clips with synchronized sound and a description. There is little such data. For editing there is even less: it requires triplets of source material, instruction and result, in both modalities at once.

The authors argue that this makes the model spend its capacity on the wrong thing. It learns to generate video and audio separately, even though huge unimodal datasets could teach that. The real goal, consistency between picture and sound, gets only leftover attention.

How the training works

UniAVC trains in two stages. First, each branch, video and audio, learns from unimodal data until it is fully competent at generation and editing. The paired data is saved for the second stage, where the model learns cross-modal correspondences: what should sound when something moves a certain way, and what a scene looks like given a certain sound.

The second half of the idea concerns noise. Video carries far more information than audio. To build spatial structure and motion, it needs a long period of high noise. The sparse audio signal resolves earlier. A single schedule for both modalities forces a compromise and hurts one of them.

A synchronized step schedule has another weakness. Scenarios where one modality serves as the condition for the other, such as adding sound to a finished video, stay untrained under that scheme. In UniAVC the schedulers are decoupled: each warps its noise schedule independently, although both start from a shared timestep. Subtasks with one clean modality directly provide a training signal for asymmetric configurations.

Key specs

• Architecture: a diffusion model with two branches, video and audio
• Training: unimodal data first, then paired data
• Noise schedule: separate for each modality, shared timestep
• Tasks: joint synthesis from text, video→audio, audio→video, instruction-based editing
• Weights: one set, no adapters for specific tasks
• Metrics, model size, clip length: not stated in the abstract

Two practical modes

Asymmetric training yields two modes of operation. The first is intent-aware editing. One modality stays untouched and acts as a constraint on the other: you can change the audio track without touching the picture, and the model uses the video as an anchor.

The second mode is self-refinement after inference. If the video came out well but the audio did not, only the problem branch is regenerated while the accepted one stays fixed. Before, this meant rerunning the whole generation or bringing in a separate sound-generation model.

What to check in the full paper

For production teams, the benefit looks like this: editing one track without rebuilding the whole clip cuts the number of full runs, and with it GPU costs. But this follows from the logic of the method, not from published measurements.

Several things are worth looking for in the paper: a comparison with models trained on paired data, an evaluation of audio-video synchronization, clip length and resolution, and parameter count. Equally important is whether the authors will release code and weights. Developers of multimodal diffusion systems should take a closer look at the technique itself: separate noise schedules with a shared timestep can carry over to other pairs of modalities whose information density differs just as sharply.