MemLearner: 學習查詢上下文記憶以用於視頻世界模型

摘要

視頻世界模型是一種互動式視頻生成模型，能根據用戶動作與歷史視頻幀預測未來世界狀態。此類模型面臨的關鍵挑戰在於缺乏記憶，導致長時間生成的場景不一致。先前的研究採用基於規則的上下文幀檢索作為記憶機制，但在場景遮擋與動態物體等情況下難以泛化。我們提出MemLearner，一種基於學習的自適應上下文查詢方法，利用查詢標記（query tokens）連結上下文標記與預測標記。透過直接運用視頻生成模型本身進行上下文查詢，MemLearner能利用預訓練的視覺先驗，無需從頭訓練額外模組，並整合高效的訓練與推論策略。我們收集了一組包含場景遮擋與動態物體的長視頻資料集，並附帶相機姿態標註，同時提出多資料集訓練策略，結合標註的渲染影片與未標註的真實世界影片。大量實驗證明，MemLearner在場景一致性與記憶能力上顯著優於先前的視頻世界模型，特別是在具有挑戰性的遮擋與動態場景中。

English

Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.