Streaming video AI almost never asks for help: the best model scores 7.22% on AskStream

A team from the Institute of Automation of the Chinese Academy of Sciences, the Beijing Academy of AI (BAAI) and two other universities has released AskStream, a test for video AI that watches a clip as if it were a live broadcast. It checks a skill that models have barely developed: realizing in time that information is missing, and asking for it. To measure this, the authors devised the Ask-F1 metric, which counts a question only if it is asked within the right time window and for the right reason. The best offline model scored 52.18%, while the best streaming model scored 7.22%. That is a sevenfold gap.

Why models need to ask

The familiar setup is that a model receives the whole video and then answers. Streaming models watch a clip as frames arrive and answer in real time. The next step is proactive models, which choose for themselves when to respond, for example by flagging an anomaly or walking a user through instructions. All of these setups assume that everything needed for an answer will eventually show up in the stream.

In practice, it doesn't. The relevant object may be blurred, the camera may be pointing the wrong way, or the user may have asked an ambiguous question. The model then either waits forever or makes up an answer. The authors propose a third action besides "wait" and "answer": ask.

They tested how useful that step is. When the key object in the video is blurred, answer accuracy was 42.3%. After the model requested a sharper video, it rose to 62.3%. Giving the model perfectly clean video from the start yields 63.1%. Irrelevant clarifications barely helped. The right information reduced the model's uncertainty and shifted probability toward the correct answer.

Eight situations where the model should ask

Uncertainty comes from two sources, the video and the user. Together they produce eight subtasks:

• KVEM: the key object in the frame is blurred, and the model should request a sharper video.
• VIM: the needed details are not visible from the current angle, and the model should ask for another one.
• ICM: an object appears in the video, but the model doesn't know its personal significance to the user (for example, whose item it is), so it has to ask.
• PACM: the user performs an action with their own name for it, and the model doesn't know the definition.
• AUQ: the question is ambiguous, and the model needs to clarify the user's intent.
• False-premise question: the model must notice that the question contradicts the video, correct the user and then answer.
• UPM: the answer depends on the user's tastes, such as style, and the model has to ask about them.
• UPC: saved preferences have gone stale, and the model must request current ones.

In some tasks the problem is visible immediately, so the model should ask and then wait for the right frames. In others, the gap only emerges as the video plays, so the question has to come later. A turn counts if the right question is asked in the right window and a correct answer follows.

How the benchmark was built and scored

The test includes 438 videos and 1,401 questions across nine domains. Clip lengths range from 16.72 seconds to 118 minutes. The videos come from YouTube and existing datasets, and some were synthesized in Blender. Object blurring was done with Grounded SAM 2, and PySceneDetect helped with scene cutting. GPT-5.6 Sol generated the answer options, and humans checked every example: without the missing information, a single unambiguous answer is impossible; with it, one is possible. In total, the test has 2,714 timestamps accurate to the frame, averaging 3.2 turns per video.

For offline models, which can't work with a stream, the authors built a hierarchical memory. The last 10 frames (1 frame per second) are stored as is. Every 10 frames are compressed into a single medium-term entry. Every four such entries are merged into a long-term one, with no more than eight long-term blocks in total. The model makes a decision once per second, with a tolerance of ±1 second. In open-ended mode, Gemini-3.8 Flash grades the answers.

What the results show

Among offline models, Gemini-3.8 Flash looks strongest: by subtask, it scores 37.17% on KVEM, 43.10% on false-premise questions and 81.36% on UPC. Results vary widely across types of uncertainty, though.

Streaming proactive models do far worse: on most subtasks they scored zero, and many failed to complete the task. The best of them, AURA, scored only 7.22%.

Errors mostly fall into two kinds: asking too early, before there is any evidence, and asking a question that misses the point. When the model is told when and what to ask, Ask-F1 rises by 43.83 percentage points. In other words, the main failure is not in answering but in recognizing the gap.

What this means for developers

For video assistants, whether for everyday tips or remote guidance during operations, this ability is practically essential. An assistant that confidently answers from a blurred frame is more dangerous than one that asks you to show it again. Streaming models aren't ready for this yet, and the benchmark gives them a measurable target. The test's drawbacks are its small scale (1,401 questions) and its heavy reliance on model judges in open-ended mode.