GameXpert-Bench：编码智能体距离专家级游戏开发还有多远？

摘要

近年来，大型语言模型（LLMs）可作为编码智能体，根据自然语言请求构建完整游戏。游戏开发尤为严苛，因为程序逻辑、视觉与音频内容、界面、交互和可玩性必须在单一可执行产物中协同运作。因此，衡量这一能力需要同时评估游戏产品和开发过程。现有基准测试往往通过评估最终产物或孤立的开发阶段来检验LLM的游戏开发能力。我们通过对完整的人机开发轨迹进行分析，识别出三个共同覆盖编码智能体游戏开发生命周期的阶段：初始游戏生成、缺陷诊断与修复，以及多轮优化。为此，我们引入GameXpert-Bench，将三个生命周期阶段具体化为三个互补的基准测试轨道。GameGen评估在空工作区中根据单一请求生成完整游戏的能力。GameFix评估在缺陷被报告或留给智能体自行发现时的诊断与修复能力。GameOpt评估通过以用户与智能体之间的真实开发轨迹为基础的请求链所实现的累积优化能力。我们通过实际游戏交互、确定性行为测试或带回归检查的最终产品标准来评估每个轨道。该基准套件包含11个游戏类型的97个生成任务；来自50个经人工验证的游戏关卡、每个关卡含19–27个注入缺陷的100个修复任务；以及包含六轮、102个请求的17条优化链。在三条轨道中，当前智能体在生成可玩的游戏基础和实现明确需求方面，比在发现缺陷、验证运行时行为以及保持跨变更的功能一致性方面更为可靠。

English

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.