UniAVC: Joint Audio-Video Synthesis via Asymmetric Noise Scheduling

Unified audio-video creation requires a model capable of text-conditioned synthesis, bidirectional cross-modal generation, and instruction-guided editing. Current systems rely on paired text-video and text-audio corpora that are rare online — and for editing tasks, nearly nonexistent.

The problem with existing approaches

Today's joint audio-video synthesis systems use paired data inefficiently. Instead of reserving scarce pairs for learning cross-modal interaction, models burn through these examples learning simple single-modality generation and editing tasks — things abundant single-modality datasets could teach just as easily.

On top of that, every existing system applies a single noise schedule across all modalities simultaneously. This ignores two critical facts: video carries far greater information density and needs an extended high-noise period to organize spatial structure and motion dynamics, while the sparser audio signal can stabilize much earlier. Synchronous denoising leaves asymmetric workflows — such as video synthesis conditioned on audio — completely untrained.

Architecture and methodology

Researchers from HKUST and collaborators developed UniAVC, which solves both problems through a two-stage training process.

In the first stage, each branch (audio and video) trains on abundant single-modality datasets until it's fully competent at generation and editing. In the second stage, paired data is devoted exclusively to learning cross-modal correspondence, letting the system link the two modalities without wasting effort on skills it has already mastered.

The second key component is modality-decoupled noise schedulers. Rather than a single noise schedule, the system independently warps the noise schedules away from a shared timestep for each modality. Subtasks involving one clean modality directly supervise asymmetric configurations, training the model to work in either direction between audio and video.

Capabilities and practical applications

These architectural choices unlock two key capabilities:

• Intent-aware editing — the system keeps one modality fully untouched, using it as a control signal to edit the other modality • Post-inference self-refinement — regenerates only the unsatisfactory branch while leaving the accepted branch unchanged • Joint synthesis — text-conditioned, simultaneous audio and video creation from a single set of weights

What sets UniAVC apart is that it covers multiple content-creation tasks without specialized adapters for each one.

The paper's authors are Kien T. Pham (HKUST, lead author), Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang, Chenyang Lei, Rui Liu, Qifeng Chen, and Long Chen. The research was published on October 17, 2026 under arXiv ID 2610.24791 and has already drawn attention from the community — its GitHub repository has racked up 247 stars. Full code and model weights are available at github.com/HKUST-LongGroup/UniAVC, with an interactive demo at hkust-longgroup.github.io/UniAVC.