Marionette：預測世界狀態、渲染幾何、繪製外觀

摘要

互動式遊戲世界模型通常直接在像素或潛在空間中自回歸視覺觀測，迫使姿態、幾何與遮擋等結構化屬性必須由同一個生成序列隱式維持。在長程時間跨度中，這些潛在世界屬性的誤差會累積，使一致性與可控性變得脆弱。我們改為明確建模不斷演化的世界狀態，將精確的幾何計算委派給一個固定且零參數的渲染器，而讓神經模型負責合成外觀。我們將此概念實現為 Marionette——一個針對具有關節角色的互動式遊戲所設計的世界模型。首先，一個兩階段自回歸動力學模型預測明確且可解釋的 276 維 3D 世界狀態，其中包含多實體關節骨骼、度量根節點軌跡與旋轉。其次，一個零參數圖形橋接將預測狀態轉換為姿態控制影片，以閉式解計算世界空間幾何與遮擋。第三，一個以控制條件化的影片擴散觀測模型，從所產生的結構化控制信號中合成照片級逼真的 RGB 觀測。我們的實驗確立了 Marionette 的兩項特性。第一，預測的世界狀態可直接控制。強制輸入不匹配的動作流，在 48 個保留測試片段上使根部對齊關節誤差改變 31%。第二，長程行為由狀態決定，且可在該狀態中修復。若任其自由發展，兩個生成角色會漂移到相距 21.2 公尺（錄製的會話維持在約 5 公尺），且三分之一的影格出現地面穿透。對明確狀態施加兩條規則——地形碰撞器與分離上限——可將穿透降低 66%，並保持兩個角色持續互動，且無需修改觀測模型。將外觀生成經由預測狀態引導，並不會造成我們所能偵測到的保真度損失；其 FVD 為 831，而錄製姿態則為 799。

English

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.