Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarMIT News→ original

Audio AI Scores Well on Tests Even With Random Noise, MIT and IBM Find

Audio models on multiple-choice tests often don't listen to the recording at all. They guess the answer from the question text. In one example from the MIT and IBM team, the question asks which animal growls, with the options "chipmunk, lion, alligator, chickadee," and the model simply picks "lion." When random noise replaces the recording, the scores remain decent. So when testing audio models, add noise and silence: if the score doesn't drop, the test isn't measuring hearing.

AI-processed from MIT News; edited by Hamidun News
Audio AI Scores Well on Tests Even With Random Noise, MIT and IBM Find
Source: MIT News. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMIT News ↗
Editor-in-Chief
Key metric #1IPLoc replaces "airplane" with the name "Elizabeth" to force the model to look at the image
Key metric #2The MIT-IBM project has run for many years, and the team has published about two dozen papers
Key metric #3Next, the team plans to tackle models that listen while they are speaking

Imagine you're choosing an audio model for a voice product and looking at its benchmark results. The score is high, so the model must understand sound well? Not necessarily. The MIT-IBM Computing Research Lab, led by Jim Glass, a senior research scientist at MIT's CSAIL, found that many audio models don't listen to the recording at all on multiple-choice tests. They guess the correct answer from the question text. When random noise replaces the sound, the results still remain decent.

A model that "listens" without sound

We're talking about models that can perceive audio and reason about it. They're most often tested in a simple way: ask a question and offer several answer options. This is cheaper than grading free-form answers. Free-form answers would have to be checked either by people, which is expensive and slow, or by another language model acting as a judge, which has its own weaknesses.

Glass gives a clear example. Question: which animal growls in the recording? Options: chipmunk, lion, alligator, chickadee. The model doesn't listen to the sound. It sees the word "lion" among the options and picks it. The researchers confirmed this by feeding random noise instead of the recording. The models still performed fairly well, which means they relied on the question text.

In the OmniR1 paper, the team proposed ways to make a model pay closer attention to sound and rely less on cues in the question. The practical takeaway for anyone evaluating audio models: include control cases with noise or silence in your evaluation. If the score barely drops, your test isn't measuring what it should. It also helps to compare a model across different types of benchmarks, not just one.

The same problem with images

A similar logic appears in work with images. The IPLoc study showed that even modern models like GPT-4o struggle to locate a specific object in a picture when given only a few examples. The authors developed a training method with name substitution: for example, the word "airplane" was replaced with the name "Elizabeth." The model can't rely on what it already knows about airplanes and has to look more carefully at the visual context.

There's another idea: a model sometimes already has the right internal representation, but it disappears after further fine-tuning. So part of the work focuses on restoring such abilities without new labeling.

How to work without a huge budget

The team admits it doesn't have the resources to train its own giant foundation model. Instead, the researchers take open models and adapt them with weakly supervised methods that need less labeled data. One example is test-time reinforcement learning. The project has no single approach: in Glass's words, "a thousand flowers bloom." Rogerio Feris, Brian Kingsbury, and Samuel Thomas from IBM Research, and Hilde Kuehne from the University of Tübingen, also took part in the work.

The project started simply: models were shown pairs of "image and sound description" as well as mismatched pairs. This way, a model learned on its own to associate the sound of the word "lighthouse" with a picture of a lighthouse, without any text. Then video and ambient sound were added: the model learned to distinguish how parrots, electric guitars, and acoustic guitars sound. Later, large language models were connected. Over the years, the team has published about two dozen notable joint papers and releases open benchmarks.

What's next and the limitations

In the coming year, the team will shift its focus to full-duplex dialogue systems. These are models that listen while they talk, the way people do, instead of waiting for their turn. Glass believes that talking with such voice AI will feel more natural. For now, open models of this type fall short of large language models: they have basic conversational skills, but they lack a sense of time and depth of knowledge.

An important caveat: this is an interview, not an article with a table of results. It contains no figures on how much the score dropped with noise. Still, the warning should be taken seriously: a high score from an audio model on a multiple-choice test doesn't yet prove that it heard anything.

Story

Key Questions Answered

What are the key details on A model that "listens" without sound?

We're talking about models that can perceive audio and reason about it. They're most often tested in a simple way: ask a question and offer several answer options. This is cheaper than grading free-form answers. Free-form answers would have to be checked either by people, which is expensive and slow, or by another language model acting as a judge, which has its own weaknesses. Glass gives a clear example. Question: which animal growls in the recording? Options: chipmunk, lion, alligator, chickadee.

What are the key details on The same problem with images?

A similar logic appears in work with images. The IPLoc study showed that even modern models like GPT-4o struggle to locate a specific object in a picture when given only a few examples. The authors developed a training method with name substitution: for example, the word "airplane" was replaced with the name "Elizabeth." The model can't rely on what it already knows about airplanes and has to look more carefully at the visual context.

How to work without a huge budget?

The team admits it doesn't have the resources to train its own giant foundation model. Instead, the researchers take open models and adapt them with weakly supervised methods that need less labeled data. One example is test-time reinforcement learning. The project has no single approach: in Glass's words, "a thousand flowers bloom." Rogerio Feris, Brian Kingsbury, and Samuel Thomas from IBM Research, and Hilde Kuehne from the University of Tübingen, also took part in the work.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…