Conversational Model Benchmark: GPT Realtime and Gemini Respond at Almost Every Pause

At every pause, a conversational model decides whether to speak or wait. Existing tests measure turn-boundary detection, rule compliance or aggregate statistics, but not this decision itself. The authors of the preprint arXiv:2610.12802 measured it directly. They put six real-time systems through 427 pauses and 482 turn endings from 176 unscripted conversations in the Seamless Interaction dataset. Cloud APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause. Mostly the full-duplex models Moshi and PersonaPlex know how to stay silent.

How the test works

Each system received the conversation history up to a chosen point. The researchers then compared what it generated with what the human did. There are several metrics, and each answers its own question:

• whether the model speaks at the pause, and how quickly;
• the latency, length and wording of its response at the end of a turn;
• its ability to answer a question about the previous 30–120 seconds of conversation.

The main difference from familiar benchmarks is that the reference is the live behavior of the human speaker. The test does not check against an abstract rule such as "a pause longer than 700 ms means it is OK to speak."

Who joins the conversation, and when

Cloud systems respond at almost every pause, whatever end-of-turn detector they use. The detector affects only latency. With silence-based detectors it is roughly 1–4 seconds; with semantic end-of-turn detection it is about 8 seconds. So whether the words land inside the pause depends only on timing. The model does not choose whether to speak at all.

The full-duplex Moshi and PersonaPlex behave differently. They are the only ones that regularly stay silent, and the only ones that cut in within human reaction time. Cloud systems also respond at greater length than the human does.

The price of good manners: memory

On questions about the earlier conversation, the picture reverses. The cloud systems received the history as text, and they recall 41–60% of facts as scored by the Claude Opus 5.5 judge. Under strict word matching the result is lower: 19–26%. The full-duplex models listen to the history as audio and recall 17–22% of facts by the judge and 12–15% by the strict matcher. In other words, a natural rhythm currently comes at the cost of memory, and the cloud systems' memory comes from the fact that they do not listen at all but read.

The conditions of the comparison also matter. The input formats differ between the two groups of models: text in one case, audio in the other. The gap therefore cannot be attributed entirely to architecture.

The authors are publishing the labels, the selection points, the test harness and every model output. The results can be re-checked, and new systems can be added to the comparison.

If you are building a voice assistant on a cloud API, look at more than average latency. Check whether the system can stay quiet at a pause where the person is simply thinking. The type of end-of-turn detector sets the trade-off: 1–4 seconds with silence-based detection versus about 8 s with a semantic detector. If memory of the conversation matters for your product, a text history is the more reliable option for now. If rhythm matters, you will have to look at full-duplex models and accept their weaker recall of the past. The dataset and harness from the release are suitable for running your own stack on real pauses rather than synthetic ones.