AMBIT trains one model to generate and edit 3D sound from text

Spatial audio is still largely assembled by hand: a sound designer decides which sources are needed, where to place them, and how they move, then synthesizes the sound field as a separate step. A team of researchers under the NoizAI banner proposes fusing these stages into a single model. The AMBIT system (arXiv 2610.34415) takes text as input and outputs native First-Order Ambisonics — a four-channel format that describes sound surrounding the listener. It works in two modes: generation from scratch and instruction-based editing. The authors are Kam Man Wu, Yu Zhang, Pengjun Fang, Mingfei Chen, Weijia Chen, Zeyue Tian, and Qifeng Chen.

A scene plan as an executable artifact

The central idea behind AMBIT is the ScenePlan. Rather than a vague text description, it's a structured plan that explicitly fixes three things: the content of each source (say, a line of dialogue, an instrument, or a noise), its activity over time, and its three-dimensional trajectory. The model first writes this plan, then renders the audio from it.

Both stages run on a shared Transfusion backbone. The autoregressive component plans the scene, while the rectified-flow component renders the sound field. Separate models for planning and synthesis inevitably drift apart in their internal representations; here, both modes operate on shared weights.

During editing, the plan describes the entire target scene, including sources that should remain unchanged. A request like "remove the guitar on the left" turns into a new, complete plan in which the guitar is gone and everything else is spelled out explicitly. Contrastive features tie the requested change to the right fragment. Full latents from the reference audio add residual acoustic detail so that anything not meant to change doesn't drift.

A codec that doesn't break the channels

Standard audio autoencoders are built for mono or stereo, but in Ambisonics the inter-channel structure matters — it's what lets the ear determine direction. That's why AMBIT uses an FOA-aware continuous VAE. It combines three techniques:

• reconstructing the omnidirectional signal through a shared decoder, to preserve content;
• asymmetric grouped KL regularization, which treats different latent groups differently;
• spatial covariance alignment, so the reconstruction retains the relationships between channels.

Without such a codec, the renderer would operate in a latent space that fails to distinguish "sounds similar" from "sounds like it's coming from the same direction."

Self-distillation on its own mistakes

The third contribution is joint on-policy self-distillation (OPSD) for Transfusion. The planner trains on prefixes generated by the student itself, rather than on perfect reference plans. The renderer trains on the sampling states it actually reaches when working from the student's real, imperfect plans. This narrows the gap between training and inference. Supervision comes from teachers that have access to the target scene, guiding both modes of the shared student — planning and rendering.

What the authors claim

Evaluation covered speech, music, and sound effects. Compared with selected baselines, AMBIT shows better content and spatial quality in generation, higher acoustic accuracy when editing by instruction, and better preservation of unmodified sources. The paper made Hugging Face Daily Papers and gathered 33 upvotes.

Key elements of the system:

• format: native First-Order Ambisonics;
• modes: generation and instruction-based editing;
• backbone: a shared Transfusion model (autoregressive planning plus rectified-flow rendering);
• interface: executable ScenePlans with source content, activity, and 3D trajectories;
• editing: full target plan, contrastive features for the change, reference latents for preservation;
• codec: FOA-aware VAE with a shared decoder, asymmetric grouped KL, and covariance alignment;
• training: OPSD with teachers that know the target scene.

Where this could be useful

The Ambisonics format is used in VR and AR, 360-degree video, games, and immersive audio, where sound needs to rotate along with the listener's head. Today, such scenes are recorded with specialized microphones or assembled in a studio. If text-to-audio generation becomes reliable enough, a draft spatial soundscape could be produced in minutes, and edits like repositioning a source or swapping out one sound wouldn't require rebuilding the whole scene.

For developers, the approach itself is valuable: an explicit plan that can be checked and corrected before rendering, plus shared weights for both generation and editing. The source code is available on GitHub, and the authors have a project page. There are no separate models or datasets on Hugging Face yet, so reproducing the results will require working from the repository.