MemoBench：在动态变化环境中评估世界建模的基准测试

摘要

视频生成模型旨在模拟动态环境，现有多个基准测试已能评估帧间记忆一致性。然而，多数基准仅在目标物体保持可见时测试一致性，少数迫使物体离开视野的基准则评估静态场景（遮挡期间无任何变化）。为填补这一空白，我们提出MemoBench——一个基于动态环境中“消失-重现”范式构建的诊断基准：目标物体经历某种物理过程后从视野中消失，需在其重现时正确恢复其更新后的状态。我们整理了涵盖合成场景与真实场景的360个真实标注片段，并设计了一套评估框架，结合自动化指标与基于视觉问答（VQA）的评估方法，从四个诊断维度展开分析。对八种当前最先进模型的评估揭示了在“消失-重现”范式下记忆一致性方面的关键见解与待解挑战。

English

Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.