NTU Scientists Teach AI Model to Skip Video: 59% Fewer Tokens

Why video is an expensive, not-always-necessary luxury

Researchers at the College of Computing and Data Science at Nanyang Technological University (Singapore) have tackled a problem holding back "omni-models" — AI systems capable of processing text, audio, images, and video simultaneously — from running fast and cheap. Models like Qwen2.5-Omni and Qwen3-Omni deliver strong results precisely because they fuse data from multiple sources into a single understanding of a situation. But that power comes at a cost: processing video is far more expensive than processing text and audio, and in practice, video often isn't even needed to get the right answer.

On the WorldSense benchmark (tests of understanding events through simultaneous audio and video), the authors measured that a text-and-audio query averages around 3,000 input tokens, while adding video to that same query pushes it to nearly 40,000 tokens — more than a 13x jump. Yet for some questions video is genuinely critical (you can't answer "what color is the door" without the footage), while for others it just gets in the way. In one example cited in the paper, Qwen2.5-Omni answered correctly using audio alone but got it wrong once video was added to the query — the extra modality threw the model off track.

How modality selection works for a given query

Rather than retraining the omni-model itself or compressing video tokens, the authors propose placing a lightweight "router" in front of the model. It looks only at the text of the question — via BGE-M3 embeddings — and, before the main model even runs, decides whether text and audio (TA) are sufficient or whether video (TAV) needs to be brought in as well.

The router isn't trained as a standard classifier — it's trained on a cost-sensitive task. For each training example, the system calculates which option (TA or TAV) would give the best balance of accuracy and cost, and that option becomes the router's correct label. Queries where picking the wrong modality is especially costly — either badly hurting accuracy or significantly driving up inference cost — get a higher weight during training. This way, the router learns to judge when video is actually needed by focusing on the examples where that judgment matters most.

Results in practice

The approach was tested on the WorldSense benchmark with two models — Qwen2.5-Omni-7B and Qwen3-Omni — measuring query cost three ways: number of input tokens, response latency, and dollar cost (based on per-modality token pricing).

• For Qwen2.5-Omni, the cost-sensitive router matched the accuracy of the "always use every modality" (TAV) mode while spending only about 41% of the input tokens, 39% of the response latency, and 61% of the dollar cost compared to the full mode.
• That translates to savings of nearly 59% in tokens, 61% in time, and 39% in money, while preserving baseline accuracy.
• The authors compared their method against several alternatives: random modality selection, a judge based on Gemini 2.5 Pro that decides from the question text whether video is needed, and cascading schemes — where the model first answers using text and audio, then falls back to video if its confidence is low. Across all three cost metrics, the proposed cost-sensitive classifier delivered a better accuracy-to-cost ratio than every alternative tested.
• An Estimate-Then-Predict method (separate models predicting the quality and cost of each modality configuration) also beat the cascading schemes, but lost out to the main proposed approach.

What this means for engineers

The practical takeaway: if you're building a product on omni-models — video analytics, content moderation, camera-equipped assistants — you don't necessarily need to feed video into every single query by default. A simple, cheap classifier based on the query text alone is enough to figure out when video is actually needed — potentially saving more than half your inference budget with no hit to answer quality. This approach is especially valuable for high-traffic services, where every extra token and millisecond of latency directly hits the inference bill.