news

A code-only video benchmark judges 2,151 agent rollouts, and its own pages give two different task counts

VibeVideoBench runs agents in a sandbox with no image or video generation model, scores five quality dimensions with three VLM judge panels, and finds that counting undelivered videos as zero reorders the ranking.

By Priya Shenoy · 29 October 2026

A video generation benchmark published this week contains no video generation model. VibeVideoBench puts multimodal agents in a sandbox for one hour, blocks image and video models outright, and asks each agent to deliver a finished MP4 with sound by writing code that renders every frame and synthesizes every sound. Across nine frontier agent systems and 2,151 scored rollouts, the strongest system satisfied every core requirement on 76.3% of tasks, and the most consequential failure was not visual quality but missing the deadline.

The premise is a list of things diffusion models get wrong

The paper, from Muxi Diao and 23 co-authors, opens with an argument rather than a leaderboard: diffusion video models produce striking clips but cannot reliably render an exact formula, a correct chart, a rule-consistent game, or narration synchronized to what is on screen. The proposed alternative is procedural. Agents build their own rendering pipelines, so a chart is correct because the code drew it correctly, not because a sampler happened to place the right pixels.

That framing lands on a weakness anyone benchmarking generated video already tracks. In our own testing, text rendering is scored as a separate category precisely because it is where models most often produce something that looks right and reads wrong.

Program checks, 13,000 rubric items, and three judge panels

The scoring is the part worth reading closely. Evaluation runs in layers: program checks of the delivery specification, roughly 13,000 task-specific rubric checks that are each traced back to the requirement they enforce, and five quality dimensions anchored per task and scored by three panels of vision-language model judges. The authors report that the judges' ranking of the nine systems closely matches blind human preferences, which is the claim most automated judging setups skip. Judge panels that are never checked against humans have a habit of agreeing with themselves and nothing else.

GPT-6 Astra and Claude Opus 5.5 take first and second place with both the judges and the human annotators. The paper also notes that agents inspect their own rendered frames, and that how much they look does not predict how good the output is.

Counting missing videos as zero changes who wins

Delivery is treated as a second axis rather than a footnote. When undelivered videos are scored as zero instead of dropped, the ranking changes. Three quarters of the lost videos were final renders that started too late, ran too slowly, or never started at all, which the authors attribute to how agents budget their hour rather than to any rendering limitation. For a benchmark built around a hard wall-clock limit, that distinction decides the board.

239 tasks in the paper, 178 on the project page

The published descriptions do not agree on the size of the suite. The arXiv abstract and the Hugging Face paper page both describe 239 self-contained tasks in English and Chinese across 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers. The benchmark's own project repository describes 178 tasks, nine agent systems, and three independent video judges. The agent count and the judge structure match; the task count does not, and neither page explains the gap.

That matters for anyone who wants to reproduce the numbers, because 76.3% of 239 and 76.3% of 178 are not the same evidence. Until the project page and the paper carry the same figure, the task count is the one number in this release that cannot be taken at face value.

Sources: arXiv, Hugging Face, VibeVideoBench project repository.

Priya Shenoy: Tracks what AI video actually costs across the platforms that resell access to the same handful of models. Treats a pricing page as a claim, not a fact, until someone checks it.