擴散模型是即時遊戲引擎。

Diffusion Models Are Real-Time Game Engines

August 27, 2024

作者: Dani Valevski, Yaniv Leviathan, Moab Arar, Shlomi Fruchter

cs.AI

摘要

我們介紹了 GameNGen，這是第一個完全由神經模型驅動的遊戲引擎，能夠以高質量在複雜環境中長時間軌跡的實時互動。GameNGen 可以在單個 TPU 上以每秒超過 20 幀的速度互動模擬經典遊戲 DOOM。下一幀預測實現了 PSNR 為 29.4，與有損 JPEG 壓縮相當。人類評分者僅略優於隨機機會來區分遊戲短片和模擬短片。GameNGen 訓練分為兩個階段：(1) 一個強化學習代理學會玩遊戲並記錄訓練過程，以及 (2) 訓練擴散模型以在過去幀和動作序列的條件下生成下一幀。條件增強使得在長軌跡上穩定自回歸生成成為可能。

English

We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.