Gemini 2.5 Flash Replaces Human Raters in Voice Agent Testing

On July 10, 2026, a group of researchers published a paper on arXiv evaluating the reliability of Gemini models as automated audio judges (LALM, Large Audio Language Model) for full-duplex voice agents. The models listen to a raw stereo recording of a conversation and assign quality scores without an intermediate transcription step.

How the experiment works

The reference model was Gemini 2.5 Flash, which the researchers compared with three calibrated human raters on a corpus of 209 stereo sessions. The corpus has two parts: 152 real full-duplex conversations across 13 strata of accent and recording conditions, and 57 clips with deliberately injected defects, designed to test the model's sensitivity to specific agent errors.

Each session is scored on 8 production quality dimensions, including speech naturalness, response timeliness, interruption handling and comprehension.

Key dataset figures:

• 209 stereo sessions, including 57 with artificial defects
• 13 strata by accent and recording conditions
• 8 quality dimensions (production dimensions)
• 3 calibrated human raters
• Gemini 3.5 Flash and Gemini 3.1 Pro were tested in parallel

How well does Gemini agree with humans?

The authors apply three independent tests, and Gemini 2.5 Flash passes them consistently.

Rank correlation: on 5 of 8 dimensions, the gap between the model-human Spearman coefficient and the human-human coefficient does not exceed 0.07. On 7 of 8 dimensions, the 95% bootstrap intervals overlap, so there is no statistically significant difference.

Simple score agreement: on 6 of 8 dimensions, the model differs from the mean score of the three humans by no more than 1 point in 60–92% of sessions.

Sensitivity to defects: in 45 of 48 cells of the "defect type × dimension" grid, the model detects deliberately injected errors at least as well as humans. The authors caution that most cells are statistically underpowered, so the result should be read as "no worse than," not "proven equivalent."

What changes when the model changes

Gemini 3.5 Flash raises simple agreement with humans to 8 of 8 dimensions, which potentially makes it better suited to automated grading.

Gemini 3.1 Pro, however, despite a similar rank correlation, systematically scores lower than humans on several dimensions. The authors draw a key conclusion: high rank correlation does not guarantee correct calibration. Switching from one model to another requires separate validation on a calibration set, and findings cannot be carried over automatically from one Gemini version to another.

What this means

The paper provides empirical grounds for replacing some human raters with Gemini in production testing of voice agents: automated grading costs roughly 100 times less than human grading, at comparable quality on most dimensions. For teams building voice AI assistants with full-duplex support, this could mean a substantial cut in quality-control costs, provided the specific model is carefully calibrated.

Frequently asked questions

How much cheaper is it to evaluate voice agents with Gemini?

According to the authors, automated LALM grading costs roughly 100 times less than human grading, at comparable quality for production testing.

What happens when

Gemini 2.5 Flash is swapped for another model in the family?

Gemini 3.5 Flash raises simple agreement with humans to 8 of 8 dimensions. Gemini 3.1 Pro, by contrast, systematically scores lower than humans on several dimensions even though its rank correlation remains comparable. The authors recommend recalibrating any new model rather than relying on ranking metrics alone.