VMem-Bench Shows: Memory in Video Generators Drops Accuracy to 0.845

Long-video generators increasingly pull past frames from history so a character doesn't change face from scene to scene. It sounds logical: the more access to the past, the more stable the future. A team of Yuzhuo Chen, Huafeng Shi, Xinyu Wang, Yulong Xu, Yucheng Wang, Haoqin Hong, Guoxin Zhang, Zehua Ma, and Weiming Zhang tested this assumption using the VMem-Bench benchmark. Memory does raise entity recall from 0.559 to 0.628–0.693. But there's a cost: precision drops from 0.977 to 0.915–0.938, and the ability to avoid drawing forbidden characters and objects falls from 0.968 to 0.845–0.898.

The image is right about the past and wrong about the present

The paper's core idea is simple: an image can precisely match what happened earlier and still be wrong for the current moment in the story. A character changed clothes, got a scar, or lost an item, and the memory system helpfully feeds the generator an outdated reference. As a result, the model reproduces an obsolete appearance that the plot has already overturned.

The authors showed that this is exactly what happens. Retrieval from history reinforces outdated appearances after the story has already changed them. Similarity to past appearances increases, and by "consistency" metrics everything looks great. The problem is that consistency relative to the past and correctness relative to the current plot measure different things.

Two tracks to separate reference selection from generator performance

VMem-Bench is structured as a two-part diagnostic study.

• Track-A evaluates which references get selected from history, before any generation and without influence from the generator itself. Data: 92 real films, 16,548 segments.
• Track-B compares four retrieval policies against two control variants on the same generator. Data: 30 causally structured stories, 2,824 segments. Controls: a no-memory variant and a "last-seen" variant with a hint from ground-truth annotation.

This split removes the main source of confusion in the field. When a video turns out well, it's unclear whether a good reference choice helped or the generator simply managed on its own. Track-A answers the first question separately.

The long-gap recall trap

There's also a reverse effect. High recall over long gaps between a character's appearances can arise without memory at all, if the prompt itself already describes the appearance. The model draws the right character because it was told to, not because it "remembered." To expose this, the authors ran an analysis removing appearance descriptions from prompts. Without this check, the gain could be mistakenly attributed to the memory mechanism.

Key numbers

• Entity recall: 0.559 without memory → 0.628–0.693 with history retrieval.
• Precision: 0.977 without memory → 0.915–0.938 with memory.
• Avoidance of forbidden entities: 0.968 without memory → 0.845–0.898 with memory.
• Track-A scale: 92 films, 16,548 segments.
• Track-B scale: 30 stories, 2,824 segments, 4 retrieval policies, 2 controls.
• Side effect: locking in outdated appearances after plot changes.

What to change in pipelines

For teams building long videos, series, ad sequences, or game cinematics, the takeaway is practical. Reference relevance and overall video consistency alone don't prove that memory is actually helping. Separate precision and "forbidden entity" metrics are needed, otherwise the degradation stays invisible.

Memory should be tied to the state of the plot, not just visual similarity: retrieval needs to know that the character already changed clothes or that an object has disappeared. Testing should include a no-memory control and a variant with appearance descriptions stripped from the prompt, otherwise you risk mistaking the prompt's work for the memory's work. The code and project page are open: the Suchenl/VMem-Bench repository on GitHub, and the paper on arXiv under number 2610.35001.