Marionette：世界状态预测、几何渲染与外观绘制

摘要

交互式游戏世界模型通常直接在像素或潜空间中对视觉观测进行自回归，迫使姿态、几何和遮挡等结构化属性由同一条生成序列隐式维持。在长时间跨度上，这些潜在世界属性的误差不断累积，使一致性和可控性变得脆弱。我们显式地对演化的世界状态进行建模，将精确的几何计算委托给固定的零参数渲染器，而让神经模型负责合成外观。我们将这一思想实例化为Marionette，一个面向含关节角色的交互式游戏的世界模型。首先，一个两阶段自回归动力学模型预测显式且可解释的276维3D世界状态，该状态包含多实体关节骨架、度量根轨迹和旋转。其次，一个零参数图形桥接将预测状态转换为姿态控制视频，以闭式形式计算世界空间几何与遮挡。第三，一个控制条件的视频扩散观测模型根据得到的结构化控制合成逼真的RGB观测。我们的实验确立了Marionette的两个特性。第一，预测的世界状态可直接控制。在48个留存片段上强制施加不匹配的动作流，会使根部对齐的关节误差改变31%。第二，长时程行为在状态中决定，并可在状态中修复。不加约束时，两个生成角色漂移到相距21.2米（录制会话中保持接近5米），且三分之一的帧出现地面穿透。对显式状态施加两条规则，即地形碰撞器和分离上限，将穿透率降低66%，并保持两个角色处于互动状态，且无需改变观测模型。经由预测状态来生成外观并未造成我们能检测到的保真度损失，其FVD为831，而记录姿态的FVD为799。

English

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.