Google's StreamVibe video agent beats other models by 10.7% on response timing and quality

A team of researchers from Google, Stanford, Oxford and several US universities has presented StreamVibe, a video agent that watches a person cook or assemble furniture and decides on its own when to speak up. Its trained model, StreamVibe-VL, chooses every second between staying silent and saying something, and beats other models by 10.7% on timing and quality of responses, meaning it picks the moment for a hint more accurately and words it better. An additional memory system adds another 12.3% to proactive interaction. The task itself remains very hard for current AI, especially on long videos.

What today's video assistants can't do

According to the authors, existing video assistants are built for clear commands such as "tell me when the water boils." They are poor at picking up unspoken intentions. Another weakness is tracking what has already been done: the assistant repeats advice or answers too late. And if the user changes the request midway through a task, the system often keeps following the original one. StreamVibe, by contrast, is meant to handle even vague requests like "help me if I run into trouble."

The agent separates three types of intent: explicit, implicit, and those that change during the work. To measure the problem, the team built a benchmark of roughly 4,100 examples based on more than 183 hours of recordings of everyday activities and software use. Gemini 2.5 Flash proposed when to help and what to say, and four human raters had to approve each example unanimously. The test has three levels: clear instructions, vague goals, and requests that change over time. In one example, a user is taking apart a toy excavator and asks for progress updates, then three minutes later switches to checking quality.

How the model works and what it showed

The researchers trained StreamVibe-VL on this data, apparently based on Alibaba's Qwen3.5-9B. Every second, the model decides whether to stay quiet or speak. The second training stage uses reinforcement learning and makes the model more restrained, so it doesn't constantly interrupt the user. A separate memory system tracks which requests are still unfulfilled and which steps are already done. The model itself still decides what to say and when.

The authors report that StreamVibe-VL is 10.7% better than other models on timing and quality of responses. The memory system adds another 12.3% to proactive interaction and, among other things, cuts unnecessary remarks. StreamVibe-VL pulls clearly ahead of the base model and other open systems, but Google's proprietary Gemini 3.8 Flash still scores higher in some tests. According to the paper, the specialized training does not degrade general video understanding.

Examples from the paper and the project page show the difference clearly. In a GIMP tutorial on turning a photo black and white, the comparison model JoyAI claims the finished image has already been saved, even though the user hasn't opened it yet. StreamVibe-VL correctly suggests the next step: first "Colors," then "Desaturate." In a video about jam-toast roll-ups, other models simply describe what's on screen, for example that the bread is being rolled out, while StreamVibe-VL advises cutting off the crust first.

Where the system breaks down

The most important part of the study is its limitations. When models only need to judge at isolated moments whether a hint is appropriate, they often succeed. But when a model processes the whole video and relies on its own earlier responses, quality falls for every system tested, because errors accumulate. The content-quality metric for responses loses up to 61.5% when the model relies on its own history. On videos longer than five minutes, models rarely choose the right moment, and they are especially bad at tracking a changed instruction over a long stretch. The memory scaffolding mostly reduces the number of unnecessary interventions rather than solving the timing problem.

The authors themselves acknowledge that real-world situations are probably more complex than the test setup. Going forward, they want to study how user preferences and the environment influence each other over time.

Context: Google has been moving toward active video analysis

In September, Google showed agentic video analysis in Gemini, where the model itself decides which scenes to examine in more detail. More recently, with Gemini 4 Argon, the company demonstrated AI tracking long chains of actions in robot videos and finding errors. StreamVibe adds the ability to understand what is happening right now and recognize when help is actually needed.

Related research points to similar weaknesses. On ProactiveBench, a test of the ability to ask clarifying questions, models facing missing visual information more often guess or refuse to answer than ask. A separate video-understanding benchmark revealed serious failures in spatial and temporal reasoning. And for autonomous agents, conversation itself can worsen results, as Microsoft researchers showed.

The practical takeaway for developers: a proactive video assistant currently has to be designed with explicit task memory and a cap on how often it intervenes. It is too early to count on a single model holding the thread over a long video.