GameXpert-Bench：編碼代理距離專業遊戲開發究竟還有多遠？

摘要

近期的大型語言模型（LLMs）可作為編程智能體，依據自然語言請求建構完整遊戲。遊戲開發尤其具有挑戰性，因為程式邏輯、視覺與音訊內容、介面、互動性與可玩性必須在單一可執行成品中協同運作。因此，衡量此能力須同時評估遊戲產品與開發過程。現有基準測試常透過評估最終成品或單一開發階段來衡量 LLM 的遊戲開發能力。我們對完整的人類與智能體開發軌跡進行分析，識別出三個階段，共同涵蓋以編程智能體進行遊戲開發的生命週期：初始遊戲生成、錯誤診斷與修復，以及多輪最佳化。為此，我們提出 GameXpert-Bench，將三個生命週期階段操作化為三個互補的基準測試軌道。GameGen 評估在空白工作區中從單一請求進行完整遊戲創作的能力。GameFix 評估在缺陷被回報或留待智能體自行發現時的診斷與修復能力。GameOpt 則透過以使用者與智能體之間的實際開發軌跡為種子之請求鏈，評估累積式最佳化。我們以即時遊戲互動、確定性行為測試或帶有回歸檢查的最終產品標準來評估每個軌道。此測試套件包含 11 個遊戲類型中的 97 個生成任務；來自 50 個經人工驗證之遊戲關卡的 100 個修復任務，每個關卡有 19 至 27 個注入的缺陷；以及 17 個最佳化鏈，共六輪、102 個請求。在三個軌道中，當前智能體在產出可遊玩的基礎與實作明確需求方面，比在發現缺陷、驗證運行時行為以及跨變更保持功能方面更為可靠。

English

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.