ReWorld：一种具有长时记忆的交互式世界模型

摘要

交互式世界模型必须跟随用户的动作、记住已展示的场景，并实时流式生成。这种张力是结构性的：控制需要短视界，记忆则需要无界视界。ReWorld在训练时将二者分离，并在推理时加以约束。混合逐头注意力窗口将大多数注意力头限制在近期，而一小部分全局头则关注全部历史；随机头路由确保任一能力不会绑定到特定注意力头上；随机分块丢弃使稀疏历史处于分布内。在推理时，全部历史在固定预算下存续：一个有界KV缓存，由姿态索引的地标库支撑，模型从中检索最接近当前姿态的地标。度量尺度对齐的数据引擎将八个数据源——虚幻引擎渲染的飞行穿越、游戏漫游和真实世界视频——置于同一物理动作尺度上，使每次按键在每个数据源中移动相机相同距离；回文轨迹提供了记忆训练所需的回访证据。限于LoRA适配器的分布匹配蒸馏将采样压缩到四步：一个主干同时服务高保真多步模式和实时交互模式，以704×1280分辨率流式生成照片级真实、游戏风格和风格化世界的视频。在涵盖动作跟随、长时程回忆和视频质量的三轴协议下，与六个最近的交互式世界模型相比，它取得了最佳的控制保真度（11.95°旋转误差和最佳的相机运动一致性）以及最佳的生成质量；在一分钟的往返展开（64秒，384个潜变量）中，其固定的12块缓存仍能重建起始视图——而此时滑动窗口早已逐出证据，全KV注意力也已内存耗尽。

English

An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity (11.95^circ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts (64\,s, 384 latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.