MemLearner：学习查询上下文记忆以用于视频世界模型

摘要

视频世界模型是一种交互式视频生成模型，能够根据用户操作和历史视频帧预测未来世界状态。该模型面临的关键挑战在于缺乏记忆能力，导致在长时程生成中场景出现不一致。以往方法探索了基于规则的上下文帧检索作为记忆机制，但在场景遮挡和动态物体存在的情况下难以泛化。本文提出MemLearner——一种基于学习的自适应上下文查询方法，通过查询令牌桥接上下文与预测令牌。借助视频生成模型自身进行上下文查询，MemLearner无需从头训练额外模块即可利用预训练的视觉先验，并整合了高效的训练与推理策略。我们构建了包含场景遮挡与动态物体的长视频数据集（配有相机位姿标注），并提出多数据集训练策略，同时利用带标注的渲染视频与无标注的真实视频。大量实验表明，MemLearner在场景一致性与记忆能力方面显著优于先前的视频世界模型，尤其是在具有挑战性的遮挡与动态场景中。

English

Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.