Gemini 2.5 Flash replaces human evaluators in voice agent testing

A group of researchers published on arXiv on July 10, 2026, a study evaluating the reliability of Gemini models as automatic audio judges (LALM, Large Audio Language Model) for full-duplex voice agents: the models listen to raw stereo recordings of conversations and assign quality scores without intermediate transcription.

How the Experiment Is Structured

Gemini 2.5 Flash was selected as the reference model and compared against three calibrated human raters on a corpus of 209 stereo sessions. The corpus is divided into two blocks: 152 real full-duplex conversations across 13 strata of accent and recording conditions, as well as 57 clips with intentionally injected defects—to test the model's sensitivity to specific agent errors.

Each session is evaluated on 8 production quality dimensions—among them speech naturalness, response timeliness, interruption, and comprehension.

Key dataset numbers:

• 209 stereo sessions, including 57 with artificial defects
• 13 strata by accent and recording conditions
• 8 quality parameters (production dimensions)
• 3 calibrated human raters
• Gemini 3.5 Flash and Gemini 3.1 Pro were tested in parallel

How Well Does Gemini Align with Humans?

The authors apply three independent tests, and Gemini 2.5 Flash passes them consistently.

Rank correlation: on 5 of 8 parameters, the difference between the Spearman coefficient "model–human" and "human–human" does not exceed 0.07. On 7 of 8 parameters, 95-percent bootstrap intervals overlap—there is no statistically significant difference.

Simple score agreement: on 6 of 8 parameters, the model diverges from the average human rater score by no more than 1 point in 60–92% of sessions.

Defect sensitivity: in 45 of 48 cells of "defect type × parameter," the model detects intentionally injected errors at least as well as humans. The authors note: most cells are statistically underpowered, so the result should be read as "no worse," not "provably equivalent."

What Changes When Switching Models

Gemini 3.5 Flash improves simple agreement with humans to 8 of 8 parameters—making it potentially better suited for automatic evaluation.

However, Gemini 3.1 Pro, despite similar rank correlation, systematically underestimates scores on several parameters compared to humans. The authors draw a key conclusion: high rank correlation does not guarantee proper calibration. When switching from one model to another, separate validation on a calibration set is required—conclusions cannot be automatically transferred from one Gemini version to another.

What This Means

The work provides empirical justification for replacing some human raters with Gemini in production testing of voice agents: the cost of automatic evaluation is approximately 100 times lower than human evaluation while maintaining comparable quality on most parameters. For teams developing voice AI assistants with full-duplex support, this can mean significant cost reduction in quality assurance—provided careful calibration of the specific model.

Frequently Asked Questions

How Much Cheaper Is It to Evaluate Voice Agents with Gemini?

According to the authors' estimate, the cost of automatic LALM evaluation is approximately 100 times lower than human evaluation while maintaining comparable quality in production testing.

What

Happens When Switching from Gemini 2.5 Flash to Another Family Version?

Gemini 3.5 Flash increases simple agreement with humans to 8 of 8 parameters. Gemini 3.1 Pro, by contrast, systematically underestimates scores on several parameters, although rank correlation remains comparable. The authors recommend recalibrating any new model rather than relying solely on ranking metrics.