WorldCycle：面向長時程影片世界模型的自驗證強化學習

摘要

互動式影片世界模型對於長程規劃與探索至關重要，然而它們飽受累積誤差之苦。諸如強化學習（RL）等後訓練方法可以改進這些模型，但它們遭遇驗證瓶頸：對於任意動作序列，並不存在真實未來狀態可用於衡量長期漂移。我們的關鍵洞見在於，可逆動作循環使此驗證成為可能：由動作序列與其反向序列組成的封閉循環必須在分析上返回初始狀態，從而提供關於長程正確性的免標註監督。以此為基礎，我們提出WorldCycle，一個自驗證的強化學習框架，從普通動作序列構建封閉動作循環及其重複執行，並優化兩個互補獎勵：強制鏡像前向與反向區段之間對稱性的空間閉合獎勵，以及對齊重複循環執行間狀態的時間一致性獎勵。這些獎勵迫使模型將動作學習為一致的狀態算子，而非記憶化的時間模式，並能自然延伸至基礎模型處理不佳的分布外複合動作循環。我們進一步發布CycleBench，一個針對複雜動作結構下狀態返回能力的診斷基準。WorldCycle將狀態返回漂移減少達44%，並將複合動作準確度較基礎模型提升近4倍，為物理基礎的世界模型提供重要根基。

English

Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.