ReWorld：具長程記憶的互動式世界模型

摘要

一個互動式世界模型必須遵循使用者的動作，記住它展示過的地方，並即時串流。這種張力是結構性的：控制需要短期視野，記憶需要無界視野。ReWorld在訓練時將兩者分離，並在推論時為其設限。混合的逐頭注意力視窗將大多數頭限制在近期，而一小組全局頭則關注整個歷史；隨機頭路由防止任一能力綁定到特定頭；隨機區塊丟棄則使稀疏歷史落在分佈內。在推論時，整個過去都受到固定預算的約束：一個由位姿索引地標庫支撐的有界KV快取，模型從中檢索與當前位姿最近的地標。一個度量尺度對齊的資料引擎將八個來源——Unreal渲染的飛越鏡頭、遊戲漫遊和真實世界影片——置於同一物理動作尺度上，因此相同的按鍵在每個來源中都能讓攝影機移動相同的距離；迴文軌跡則提供了記憶訓練所需的重訪證據。僅限於LoRA適配器的分佈匹配蒸餾將取樣壓縮為四步：單一骨幹網路同時支撐高保真多步模式與即時互動模式，在照片級寫實、遊戲風格和風格化世界中串流704×1280影片。在涵蓋動作跟隨、長時域回憶與影片品質的三軸協議下，與六個近期的互動世界模型相比，它達到了最佳控制保真度（11.95°旋轉誤差與最佳攝影機運動一致性）以及最佳生成品質；在長達一分鐘的去回推演（64秒，384個潛在變量）中，其固定12區塊快取仍能重新生成起始視圖——而在這種推演長度下，滑動視窗早已將證據逐出，完整KV注意力也耗盡了記憶體。

English

An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity (11.95^circ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts (64\,s, 384 latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.