VibeVideoBench: Agents Write Videos as Code, and the Best One Handles 76.3% of Tasks

Diffusion video models produce striking clips, but they cannot reliably draw an exact formula, a correct chart, a game that follows its rules, or a voiceover synchronized with events on screen. An alternative has emerged: multimodal agents write code that renders every frame and synthesizes every sound. The authors of VibeVideoBench, including Muxi Diao, Wenbin Xu, Lele Yang and Zhanyu Ma, built what they say is the first open-domain benchmark with audio for such agents. It includes 239 tasks, nine frontier systems and 2,151 scored runs. The best system meets all key requirements in only 76.3% of tasks.

One hour, a sandbox and only TTS

The tasks are self-contained and written in English and Chinese. They fall into 12 categories: data stories, music-driven clips, school lessons, breakdowns of scientific papers, games, build-process videos and others. The agent works fully autonomously in its own "native" harness inside a sandbox. There are no image or video generators there; only speech synthesis is available. The result is a finished MP4 with sound, which must be delivered within one hour.

This tests several skills at once: choosing a rendering tool, writing a script, syncing audio with visuals and staying within the time limit. Each system uses its own rendering stack, and the authors do not impose a single pipeline.

Three-level evaluation: from the file to taste

The evaluation is hierarchical and evidence-based:

• L1. Programmatic checks of the delivery spec: the file exists, the format is correct, audio is present.
• L2. About 13 thousand rubric checks written for specific tasks. Each is tied to the requirement it verifies.
• L3. Five quality dimensions, anchored separately for each task.

Three independent panels of VLM judges do the scoring. Only judges that can hear receive the audio checks.

Who leads and where videos get lost

The ranking names three of the nine systems:

• GPT-6 Astra — first place in both the judges' scores and the blind human ratings.
• Claude Opus 5.5 — second place in both rankings.
• Gemini 4 Argon — delivers 77% of videos. If missing videos count as zero, it drops from 4th place to 7th.

Delivery becomes a second axis of the ranking. Of the 224 runs with no playable video, 169 started the final render too late, ran it too slowly or never launched it at all. The problem is how agents plan their hour, not their ability to write code. A model can build a scene brilliantly and still run out of time to render it.

Another observation: 95% of runs view their own frames, but the correlation between how much an agent looks and its final quality is practically zero (ρ = −0,07). Visual self-checking alone does not help; what matters is what the agent does with what it sees.

How far to trust the judges

VLM judges rank systems the same way humans do (ρ = 0,98). But at the level of individual pairs of videos, they agree with humans in only 73% of cases. Automated scoring works for comparing agents with each other, and it is a poor fit for a verdict on a single video.

What to do with this

For teams building agents for educational, analytical or marketing video, the benchmark offers a clear reference point. Exact formulas, charts and voice synchronization are achievable with code, while diffusion is still weak here. The main failure point is time management: teams should budget for rendering in advance, start a draft render early and keep a fallback delivery path. When choosing a model for such a scenario, look at the share of videos delivered, not only at the quality of the best ones. For your own evaluations, it makes sense to use VLM judges to compare systems and to keep humans for checking specific videos. The project and data are available at vibevideobench.github.io.