ServiceNow unveils EVA, a new framework for evaluating voice AI agents

ServiceNow has unveiled EVA, an end-to-end evaluation framework for voice AI agents that tries to measure not only whether a task was completed, but also how convenient the conversation was for the human. The project was published on the Hugging Face blog on March 24, 2026, together with an open dataset, code and first results for 20 systems.

Why older tests fall short

Most existing voice AI benchmarks test individual pieces of the system one at a time: speech recognition, synthesis quality, turn-taking timing or the ability to call tools. In practice, that is not enough. A user does not talk to an STT, TTS or LLM component separately. They talk to a single agent that has to understand the request, keep context, call tools correctly and finish the task without confusion in a live dialogue.

That is why the EVA authors propose treating a voice agent as one complete product. In a phone scenario, even a small error quickly ruins the whole experience: a misheard confirmation code makes the model's good logic useless, a long list of options is hard to follow by ear, and an extra pause makes people ask again or abandon the call. Older metrics often miss such failures because they evaluate components in isolation, outside the overall user scenario.

How EVA works

EVA is built as an end-to-end check of a multi-step spoken conversation. The system simulates a real call between a voice agent and a user bot that acts according to a set goal and role. The agent must use tools, follow the scenario's rules and reach a verifiable final state.

In the initial version, the authors released a synthetic airline dataset with 50 scenarios and 15 tools, covering everything from flight rebooking to cancellations, standby and vouchers for passengers.

• The user simulator sets the caller's goal, behavior and manner of speech
• The voice agent is tested in a real audio stream
• The tool executor returns deterministic responses and changes the state of the scenario database
• Validators filter out low-quality runs without manual labeling
• A set of metrics analyzes the conversation recording, the transcript and the tool-call logs

EVA has two main overall scores. EVA-A covers accuracy: whether the agent reached the correct outcome, whether it invented a policy, and whether it distorted important entities such as a flight number or an amount. EVA-X covers user experience: whether a response was short enough for a spoken channel, whether the conversation moved forward without repetition, and whether the agent spoke at the right moment.

The authors also compute pass@3 and pass^3, to show not only the best run but also how consistently the agent behaves across several attempts at the same scenario.

What the tests showed

The team ran 20 systems through EVA, proprietary and open-source, cascaded and audio-native, and reached a main conclusion: there is a persistent trade-off between accuracy and conversation quality. No configuration dominates on both axes. Some agents complete the task better but make the conversation less convenient; others sound more pleasant but make more mistakes at critical steps and in long multi-turn scenarios. This makes model comparison noticeably fairer than a plain binary pass/fail.

"Agents that perform better at task completion often deliver a worse

user experience, and vice versa."

Another notable failure involves named entities. A single misrecognized letter in a confirmation code or flight number can break authentication and bring down the whole scenario. The authors also note that multi-step operations prove especially hard, for example when a flight must be rebooked while keeping extras such as baggage and seats.

The gap between pass@3 and pass^3 was large for many systems: an agent can solve a task once but not do so consistently. It also matters that the current release is limited to English-language airline scenarios, so expansion to noisy conditions, accents, other languages and new domains is still ahead.

What this means

The voice agent market is shifting from polished demos to tougher engineering validation. If EVA or similar frameworks catch on, the winners will not be systems that merely sound natural, but those that are accurate, concise and repeatably bring a conversation to a result in real scenarios, not only in lucky one-off runs. For enterprise deployments, this shift is especially important.