AgentSWE：编码智能体能否构建你实际想要的智能体？

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

October 9, 2026

作者: Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu

cs.AI

摘要

随着软件生产本身交到软件手中，编码智能体正越来越多地受托构建适配用户自身工作流的智能体。这种委托隐含着一个未言明的假设：交付的智能体将忠实服务于其构建时所针对的需求，而非仅仅表面上如此。现有基准均未检验这一假设。我们提出 AgentSWE，一个面向智能体软件工程的基准，涵盖智能体开发生命周期的三个阶段：根据自然语言需求创建完整智能体、编辑生产环境中的智能体代码库，以及面向可度量目标的优化。其 25 个任务中的每一个都被设定为一项委托而非一场考试：构建者依据少量示例用例进行迭代，而验收时则让冻结的提交在旨在覆盖完整需求的留出用例上运行，并在硬性通过/不通过门槛下，依据由需求导出的评分标准对行为进行评分；对于优化任务，则依据源基准自身的目标进行评分。我们在多种模型和运行框架下评估了一系列编码智能体，发现没有一个能接近交付需求所要求的结果：构建者所展示的与其实际交付之间的差距是常态，而非罕见个例。对失败轨迹的伴随实证分析识别出反复出现的失败模式，其中包括对可见用例的过拟合、看似完整的代码背后的空洞运行时，以及无依据的完成声明；每一种都是信号，编码智能体可借此学习构建所要求的智能体。

English

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.