WorldCycle：面向长时域视频世界模型的自验证强化学习

摘要

交互式视频世界模型对于长时程规划与探索至关重要，然而它们饱受复合误差之苦。强化学习（RL）等后训练方法可以改进这些模型，但会遇到验证瓶颈：对于任意的动作序列，不存在可用来衡量长期漂移的 ground-truth 未来状态。我们的关键洞察在于，可逆动作循环使得这种验证成为可能：由动作序列与其逆序列复合而成的序列必须分析性地返回初始状态，从而为长时程正确性提供无需标注的监督信号。基于此，我们提出了 WorldCycle，一个自验证的强化学习框架，该框架从普通动作序列中构造封闭动作循环及其重复执行，并优化两个互补的奖励信号：一个空间闭合奖励，强制对称的前向与反向片段之间保持对称性；以及一个时间一致性奖励，对齐跨多次循环执行的状态。这些奖励迫使模型将动作学习为一致的状态算子，而非记忆化的时间模式，并且能够自然地泛化到基础模型处理不佳的分布外复合动作循环。我们进一步发布了 CycleBench，一个针对复杂动作结构下状态返回能力的诊断基准。WorldCycle 将状态返回漂移最多降低了 44%，并将复合动作准确率相较于基础模型提升了近 4 倍，为物理上有依据的世界模型奠定了重要基础。

English

Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.