Google taught its StreamVibe video agent to decide when to step in on its own: +10.7%

A team of researchers from Google, Stanford, Oxford and several US universities has introduced StreamVibe, a video agent that decides on its own when to step in. It watches a person through a video stream, for example while they cook or take apart a toy, and speaks only when help is actually needed. The headline number of the work is 10.7%. That is how much StreamVibe-VL outperforms other models on a combination of two measures: how well-timed its prompts are and how useful they are. But the study also shows the flip side: none of the systems tested can yet hold the thread of a task over a long stretch.

How a video assistant differs from an ordinary one

Existing video assistants are built for clear commands such as "tell me when the water boils." They struggle with three things that are obvious to a human:

• unspoken intent: a person asks "help me if I run into trouble" without naming specific steps; • memory of what has been done: the assistant repeats advice or comes in late because it does not remember what has already happened; • a request that changes midway: the user changes their mind, but the model sticks to the original instruction.

An example from the paper: a person takes apart a toy excavator and asks to be told about progress, then three minutes later asks the assistant to check the quality of the assembly. The assistant has to notice the change of task and switch.

How the test was built and the model trained

To measure the problem, the authors compiled a benchmark (a set of test tasks) of roughly 4100 examples. The material comes from more than 183 hours of video of everyday activities and recordings of people working in software. Gemini 2.5 Flash proposed when to help and what to say. Four human raters then reviewed every example, and it was accepted only with unanimous approval. The test has three levels: clear instructions, vague goals, and requests that change over time.

StreamVibe-VL was trained on this data. According to the source's assumption, it is built on Alibaba's open Qwen3.5-9B model. Every second, the model decides whether to stay quiet or speak. The second stage is reinforcement learning, that is, rewards and penalties for behavior. It makes the model more restrained, so that it does not interrupt the person at every opportunity.

A separate memory system runs alongside it. It keeps a list of outstanding requests and marks completed steps. The decision whether to speak, and what to say, stays with the model itself.

What the numbers show and where it breaks down

According to the authors, StreamVibe-VL clearly beats its base model and other open models. The memory system adds another 12.3% to proactivity and, among other things, cuts unnecessary interventions. The specialized training does not degrade general video understanding. But Google's proprietary Gemini 3.8 Flash still scores higher in several tests.

The main limitation emerged when the models were tested on long videos:

• when the model only needs to judge, at isolated moments, whether a prompt is appropriate, it often gets it right; • when it watches the whole video and relies on its own earlier answers, errors accumulate and quality drops for every system tested; • the key metric for the content of responses drops by as much as 61.5% because of the system's own history; • in videos longer than five minutes, the models rarely pick the right moment and are especially poor at tracking a changed instruction.

The examples in the paper make the difference clear. In a tutorial on converting a photo to black and white in GIMP, the competing model JoyAI claimed the finished image had already been saved, even though the user had not opened it. StreamVibe-VL suggested the next step at the right moment: first "Colors," then "Desaturate." In a recipe for jam toast roll-ups, other models simply described what was happening on screen, for instance that the bread was being rolled out. StreamVibe-VL advised cutting off the crust first.

What this means for developers

The authors acknowledge that real-world situations are more complex than lab ones, and they want to study in future how user preferences and the surrounding environment affect each other. For anyone building assistants for smart glasses, training software or support tools, the takeaway is practical. A separate task memory helps, but it does not solve the problem entirely. Error accumulation over long sessions remains the main bottleneck.

The work continues a line of research at Google. In September, the company showed agentic video analysis in Gemini, where the model itself decides which scenes to examine more closely. More recently, Gemini 4 Argon showed how AI can track long sequences of actions in robot videos and find errors. Others have related results too: on the ProactiveBench test, models often guess or refuse to answer instead of asking a clarifying question. Microsoft researchers showed that for autonomous agents, the dialogue itself can reduce the quality of their work.