EvaOmni Benchmark Tests 24 Omni Models: Leader Scores 79.4% Accuracy, but Only 35.1% on Grounding

The authors of the EvaOmni benchmark propose evaluating omni models not by a single chosen answer option, but by the entire reasoning chain. The set contains 392 videos and 1027 question–answer pairs, annotated manually by humans. A model must return an answer along with the time intervals in the audio and in the video that it relied on. The authors tested 24 systems: 7 with open weights and 17 proprietary. Gemini3.7-Flash posted the best result with 79.4% accuracy, but scored only 35.1% on the Bottleneck Grounded Score metric.

Why the Benchmark Needed Evidence

Most audiovisual reasoning tests are built on multiple-choice questions. A model gets a point for the correct letter, and that's it. Such scoring doesn't show whether the model found the right sound and the right frame or simply guessed from the wording of the question. The authors point to a second problem: text-only and single-modality shortcuts. If an answer can be obtained by looking only at the picture or by reading only the question, the benchmark is not measuring what it claims to measure.

For this reason, EvaOmni uses a multi-stage annotation process. Experts first mark the audio and visual evidence and build distractors based on the video itself. Humans then check whether each question meets strict criteria. Models apply the final filter: they remove questions that can be solved from the text or from a single modality. The questions cover 15 abilities, grouped into three levels: search and localization, basic understanding, and complex reasoning.

How Grounding Is Scored

Besides the answer, the model returns the intervals of the supporting fragments. To score them, the authors introduced two metrics. Coverage Grounded Score (C.G.) shows what share of the expert-annotated evidence the model found. Bottleneck Grounded Score (B.G.) penalizes poor temporal localization in either modality: a weak spot in the audio or in the video drags the score down. Both metrics are computed together with answer correctness.

Key figures for the leader, Gemini3.7-Flash:

• overall accuracy: 79.4%;
• C.G.: 70.8%;
• B.G.: 35.1%;
• Level 3 (complex reasoning): accuracy 79.1%, B.G. 12.3%.

Gemini3.7-Flash led both open and closed competitors on all three overall metrics. But the gap within its own results is more striking than the gap between models. On hard tasks it answers correctly almost as often as it does on average across the set, yet its time intervals match the experts' in only a small share of cases.

This gap can be interpreted in different ways. The model may understand the scene as a whole but be unable to pinpoint exactly where in the stream the relevant sounds and frames are. There may also be residual shortcuts to the answer that the filtering missed. The paper gives aggregate figures, so it is impossible to tell from them which cause dominates. The authors themselves draw a cautious conclusion: even the strongest omni models still have room to grow in grounded audiovisual reasoning. Scale is also worth keeping in mind: 1027 questions is a compact set, and B.G. is by design a very strict metric.

What to Change in Evaluation and Products

For teams building products on video and audio (call analysis, camera footage review, support based on video instructions), the takeaway is practical. Multiple-choice accuracy is a poor predictor of whether a model can be trusted when its claim needs to be verified. If an answer goes into a report, a legally significant document or an operator's decision, require the model to provide timecodes separately for speech and for visuals, and check them at least on a sample.

You can build a mini version of the same scheme on your own data: 50–100 annotated examples with intervals, scoring of both the answer and the localization, and a separate calculation for the weakest modality. Such a test will quickly show where a model answers confidently but cannot show where it got the answer. In the interface, it is worth displaying a link to the fragment, so that a person can verify the output in a few seconds instead of rewatching the whole video.