GameWAM：面向電子遊戲的世界動作模型

摘要

現代電子遊戲結合了第一人稱感知、快速視覺變化、持續的世界狀態，以及異質的原生控制。現有的遊戲代理直接將視覺與任務情境映射為動作，但缺乏明確的世界動態建模；而互動式遊戲世界模型雖可根據提供的動作預測視覺未來，但並不能作為任務策略。世界-動作模型（WAMs）統一了這些目標，但在電子遊戲的動態性與開放式互動下仍 largely 未被探索。我們提出 GameWAM，據我們所知，這是第一個用於原生閉環遊戲遊玩與 GUI 控制的 WAM。GameWAM 透過並行的視覺與動作生成過程，結合區塊因果條件化與流匹配，共同生成未來的視覺觀察與可執行的鍵盤-滑鼠軌跡。為支援世界-動作聯合學習，我們建構了同步的遊戲遊玩與 GUI 軌跡。為處理異質的原生控制，GameWAM 在每個動作步驟預測遊戲遊玩/GUI 模式，並以特定模式的預測分佈與連續動作正規化生成動作。對於長時程互動，區塊循環控制會預測超出已承諾的時域範圍，僅執行短動作前綴，並根據新的觀察重新規劃；同時，循環內部的細粒度上下文與跨循環的階層式歷史保持了時間連續性。實驗結果顯示，與對比代理相比，我們的方法在執行較少原生動作的情況下，仍可達到具有競爭力的任務成功率。我們進一步揭示了低頻動作來源印記（LASI）現象：在固定條件下，所取樣動作來源的低頻成分會系統性地引導粗略的生成攝影機運動，這揭示了生成式控制中一個對來源敏感的失敗模式。專案頁面可於 https://yunncheng.github.io/GameWAM/ 查看。

English

Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/.