GameWAM：一种用于视频游戏的世界行动模型

摘要

现代视频游戏融合了第一人称感知、快速视觉变化、持久世界状态和异构原生控制。现有游戏智能体直接将视觉和任务上下文映射到动作，但缺乏显式世界动态建模；而交互式游戏世界模型能从给定动作预测视觉未来状态，却不能作为任务策略。世界-动作模型（WAMs）统一了这两个目标，但在视频游戏的动态性和开放式交互下仍鲜有探索。我们提出了GameWAM，据我们所知这是首个用于原生闭环游戏和GUI控制的世界-动作模型。GameWAM通过并行视觉与动作生成过程，结合块因果条件化和流匹配，联合生成未来视觉观测和可执行的键盘-鼠标轨迹。为支持联合世界-动作学习，我们构建了同步的游戏与GUI轨迹。为处理异构原生控制，GameWAM在每个动作步预测游戏/GUI模式，并使用模式特定的预测分布和连续动作归一化生成动作。针对长时程交互，块循环控制在已提交的预测范围之外进行预测，仅执行短动作前缀，并根据新观测重新规划，同时周期内的细粒度上下文和跨周期的层次化历史保持了时间连续性。实验表明，与对比智能体相比，GameWAM以更少的执行原生动作实现了具有竞争力的任务成功率。我们进一步发现了低频动作源印记（LASI）现象，即在固定条件化下，采样动作源的低频分量会系统性引导粗略生成的相机运动，揭示了生成式控制中的一种源敏感性失效模式。项目页面见https://yunncheng.github.io/GameWAM/。

English

Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/.