AMBIT Teaches a Single Model to Generate and Edit 3D Audio from Text

Spatial audio is still assembled by hand. A sound designer decides which sources are needed, where to place them and how they move, then synthesizes the sound field in a separate step. A team of researchers under the NoizAI banner proposes merging these stages into one model. The system, AMBIT (arXiv 2610.34415), takes text and produces native First-Order Ambisonics, a four-channel format that describes the sound around a listener. It works in two modes: generation from scratch and instruction-based editing. The authors are Kam Man Wu, Yu Zhang, Pengjun Fang, Mingfei Chen, Weijia Chen, Zeyue Tian and Qifeng Chen.

The scene plan as an executable artifact

AMBIT's central idea is the ScenePlan. It is not a loose text description but a structured plan that explicitly records three things: the content of each source (for example, a line of dialogue, an instrument or noise), its activity over time, and its three-dimensional trajectory. The model first writes such a plan, then renders audio from it.

Both stages run on a shared Transfusion backbone. The autoregressive part plans the scene, and the rectified-flow part renders the sound field. Separate models for planning and synthesis inevitably drift apart in their representations; here, both modes run on shared weights.

In editing, the plan describes the entire target scene, including the sources that must stay. A request such as "remove the guitar on the left" becomes a new, complete plan in which the guitar is gone and everything else is spelled out explicitly. Contrastive features tie the requested change to the right segment. Full latents of the reference audio add residual acoustic detail, so that whatever was not meant to change does not drift.

A codec that doesn't break channels

Conventional audio autoencoders are built for mono or stereo, but in Ambisonics the inter-channel structure is what matters: the ear uses it to determine direction. AMBIT therefore uses an FOA-aware continuous VAE. It combines three techniques:

• reconstruction of the omnidirectional signal through a shared decoder, so that content is preserved;
• asymmetric grouped KL regularization, which treats groups of latents differently;
• spatial covariance matching, so that the reconstruction keeps the relationships between channels.

Without such a codec, the renderer would operate in a latent space that cannot tell "sounds similar" from "sounds like it comes from the same direction."

Self-distillation on its own mistakes

The third contribution is joint on-policy self-distillation (OPSD) for Transfusion. The planner trains on prefixes generated by the student itself, not on ideal reference plans. The renderer trains in the sampling states it actually reaches when given the student's real, imperfect plans. This narrows the gap between training and inference. Supervision comes from teachers with access to the target scene. They guide both modes of the shared student, planning and rendering.

What the authors claim

The evaluation covers speech, music and sound effects. Against the chosen baselines, AMBIT shows better content and spatial quality in generation, higher acoustic fidelity in instruction-based editing, and better preservation of unchanged sources. The paper appeared on Hugging Face Daily Papers and collected 33 upvotes.

Key elements of the system:

• format: native First-Order Ambisonics;
• modes: generation and instruction-based editing;
• backbone: shared Transfusion (autoregressive planning plus rectified-flow rendering);
• interface: executable ScenePlans with source content, activity and 3D trajectories;
• editing: a full target plan, contrastive features for the change, reference latents for preservation;
• codec: FOA-aware VAE with a shared decoder, asymmetric grouped KL and covariance matching;
• training: OPSD with teachers that know the target scene.

Where this could be useful

Ambisonics is used in VR and AR, 360-degree video, games and immersive audio, where sound has to rotate with the listener's head. Today such scenes are recorded with specialized microphones or assembled in a studio. If text-to-audio generation becomes reliable, a rough spatial soundscape could be produced in minutes, and edits such as moving a source or replacing a single sound would not require rebuilding the entire scene.

The approach itself is useful for developers: an explicit plan that can be checked and corrected before rendering, plus shared weights for generation and editing. The source code is on GitHub, and the authors have a project page. There are no separate models or datasets on Hugging Face yet, so reproducing the results will mean working from the repository.