Audio AI Scores Well on Tests Even With Random Noise, MIT and IBM Find
Audio models on multiple-choice tests often don't listen to the recording at all. They guess the answer from the question text. In one example from the MIT and IBM team, the question asks which animal growls, with the options "chipmunk, lion, alligator, chickadee," and the model simply picks "lion." When random noise replaces the recording, the scores remain decent. So when testing audio models, add noise and silence: if the score doesn't drop, the test isn't measuring hearing.
AI-processed from MIT News; edited by Hamidun News
Imagine you're choosing an audio model for a voice product and looking at its benchmark results. The score is high, so the model must understand sound well? Not necessarily. The MIT-IBM Computing Research Lab, led by Jim Glass, a senior research scientist at MIT's CSAIL, found that many audio models don't listen to the recording at all on multiple-choice tests. They guess the correct answer from the question text. When random noise replaces the sound, the results still remain decent.
A model that "listens" without sound
We're talking about models that can perceive audio and reason about it. They're most often tested in a simple way: ask a question and offer several answer options. This is cheaper than grading free-form answers. Free-form answers would have to be checked either by people, which is expensive and slow, or by another language model acting as a judge, which has its own weaknesses.
Glass gives a clear example. Question: which animal growls in the recording? Options: chipmunk, lion, alligator, chickadee. The model doesn't listen to the sound. It sees the word "lion" among the options and picks it. The researchers confirmed this by feeding random noise instead of the recording. The models still performed fairly well, which means they relied on the question text.
In the OmniR1 paper, the team proposed ways to make a model pay closer attention to sound and rely less on cues in the question. The practical takeaway for anyone evaluating audio models: include control cases with noise or silence in your evaluation. If the score barely drops, your test isn't measuring what it should. It also helps to compare a model across different types of benchmarks, not just one.
The same problem with images
A similar logic appears in work with images. The IPLoc study showed that even modern models like GPT-4o struggle to locate a specific object in a picture when given only a few examples. The authors developed a training method with name substitution: for example, the word "airplane" was replaced with the name "Elizabeth." The model can't rely on what it already knows about airplanes and has to look more carefully at the visual context.
There's another idea: a model sometimes already has the right internal representation, but it disappears after further fine-tuning. So part of the work focuses on restoring such abilities without new labeling.
How to work without a huge budget
The team admits it doesn't have the resources to train its own giant foundation model. Instead, the researchers take open models and adapt them with weakly supervised methods that need less labeled data. One example is test-time reinforcement learning. The project has no single approach: in Glass's words, "a thousand flowers bloom." Rogerio Feris, Brian Kingsbury, and Samuel Thomas from IBM Research, and Hilde Kuehne from the University of Tübingen, also took part in the work.
The project started simply: models were shown pairs of "image and sound description" as well as mismatched pairs. This way, a model learned on its own to associate the sound of the word "lighthouse" with a picture of a lighthouse, without any text. Then video and ambient sound were added: the model learned to distinguish how parrots, electric guitars, and acoustic guitars sound. Later, large language models were connected. Over the years, the team has published about two dozen notable joint papers and releases open benchmarks.
What's next and the limitations
In the coming year, the team will shift its focus to full-duplex dialogue systems. These are models that listen while they talk, the way people do, instead of waiting for their turn. Glass believes that talking with such voice AI will feel more natural. For now, open models of this type fall short of large language models: they have basic conversational skills, but they lack a sense of time and depth of knowledge.
An important caveat: this is an interview, not an article with a table of results. It contains no figures on how much the score dropped with noise. Still, the warning should be taken seriously: a high score from an audio model on a multiple-choice test doesn't yet prove that it heard anything.
- Nov 21, 2026MLPerf Training Adds Nvidia Rubin and TPU Ironwood in First Appearance: 153 Systems
- Nov 21, 2026Waymo pulls robotaxis off highways after new entries into work zones
- Nov 21, 2026Google Opens Pre-Orders for Gemini Smart Glasses from $395, Shipping December 10
- Nov 21, 2026Space Data Centers Are Still Unprofitable: Launch Costs Run 3–19 Times Over the Break-Even Point
Key Questions Answered
What are the key details on A model that "listens" without sound?
We're talking about models that can perceive audio and reason about it. They're most often tested in a simple way: ask a question and offer several answer options. This is cheaper than grading free-form answers. Free-form answers would have to be checked either by people, which is expensive and slow, or by another language model acting as a judge, which has its own weaknesses. Glass gives a clear example. Question: which animal growls in the recording? Options: chipmunk, lion, alligator, chickadee.
What are the key details on The same problem with images?
A similar logic appears in work with images. The IPLoc study showed that even modern models like GPT-4o struggle to locate a specific object in a picture when given only a few examples. The authors developed a training method with name substitution: for example, the word "airplane" was replaced with the name "Elizabeth." The model can't rely on what it already knows about airplanes and has to look more carefully at the visual context.
How to work without a huge budget?
The team admits it doesn't have the resources to train its own giant foundation model. Instead, the researchers take open models and adapt them with weakly supervised methods that need less labeled data. One example is test-time reinforcement learning. The project has no single approach: in Glass's words, "a thousand flowers bloom." Rogerio Feris, Brian Kingsbury, and Samuel Thomas from IBM Research, and Hilde Kuehne from the University of Tübingen, also took part in the work.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.