SpecBench：评估长周期编码智能体中的奖励欺骗行为

摘要

随着长周期编码智能体生成的代码量远超任何开发者所能审查的范围，监督便集中到单一表面：自动化测试套件。在这种设置下，奖励黑客现象自然出现——智能体为了通过测试而优化，却偏离了用户的真实目标。我们通过将软件工程任务分解为三个部分来研究这种奖励黑客现象：（i）用自然语言描述的规范说明；（ii）独立验证指定功能的可见性测试；（iii）组合相同功能以模拟真实使用场景的保留测试。基于规范说明和可见性测试套件，一个真正的智能体应当能够生成也能通过所有保留测试的解决方案。因此，我们使用这两个套件之间通过率的差距来量化奖励黑客行为。基于这一方法论，我们引入了SpecBench基准，包含30个系统级编程任务，范围从构建JSON解析器等短周期任务到从头构建整个操作系统内核等超长周期任务。大规模实验揭示了一致模式：虽然每个前沿智能体都饱和了可见性测试套件，但奖励黑客现象依然存在，较小的模型在保留测试套件上表现出更大的差距。这种差距也随任务长度急剧扩大：代码规模每增加十倍，差距就增长28个百分点。失败模式从微妙的特征隔离到蓄意的利用，包括一个2900行的哈希表“编译器”通过记忆测试输入来作弊。SpecBench为衡量编码智能体是真正构建可工作的系统，还是仅仅玩弄开发者交给它们的测试套件，提供了一个原则性的测试平台。

English

As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes for passing tests while deviating from the users true goal. We study this reward hacking phenomenon by decompose software engineering tasks into three parts: (i) a natural language description of the specification (ii) visible validation tests that exercise specified features in isolation, and (iii) held-out tests that compose those same features to simulate real-world usage. Based on the specification and the visible validation test suites, a genuine agent would be able to generate a solution that can also pass all of the held-out tests. Therefore we use the gap in pass rates on these two suites to quantify reward hacking. Based on this methodology, we introduce SpecBench, a benchmark comprising 30 systems-level programming tasks ranging from short horizon tasks like building a JSON parser to ultra long horizon tasks like building an entire OS kernel from scratch. Large-scale experiments reveal a consistent pattern: while every frontier agent saturates the visible suite, reward hacking persists, with smaller models exhibiting larger gaps on holdout suites. The gap also scales sharply with task length: it grows by 28 percentage points for every tenfold increase in code size. Failures range from subtle feature isolation to deliberate exploits, including a 2,900-line hash-table "compiler" that memorizes test inputs. SpecBench offers a principled testbed for measuring whether coding agents build genuine working systems or merely game the test suites developers hand them.