AgentSWE：程式編寫代理能否打造你真正想要的代理？

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

October 9, 2026

作者: Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu

cs.AI

摘要

隨著軟體生產轉入軟體自身手中，編碼代理越來越常受委託打造貼合使用者自身工作流程的代理。這種委派帶有一個隱而未宣的假設：交付的代理會忠實服務其被建構所欲滿足的需求，而不只是表面上如此。現有基準皆未檢驗此假設。我們提出 AgentSWE，一個涵蓋代理開發生命週期三階段的代理軟體工程基準：依自然語言需求創建完整代理、編輯生產級代理程式碼庫，以及朝向可量測目標進行最佳化。其 25 項任務皆以委託而非考試的形式設計：建構者針對少數示例案例反覆迭代，而驗收則以留出案例測試凍結的提交，這些案例旨在涵蓋完整需求；評分方式是在嚴格通過／失敗閘門下，依據需求衍生的評分規準評估行為，若為最佳化，則依來源基準自身的目標評分。我們評估了跨模型與執行框架的多種編碼代理，發現沒有任何一個能接近交付需求所要求之物：建構者所展示的與實際交付之間的落差是常態，而非特例。隨附的失敗軌跡實證分析辨識出反覆出現的失敗模式，其中包括對可見案例的過度擬合、看似完整程式碼背後的空殼執行環境，以及缺乏根據的完成宣稱；每一項都是編碼代理可藉以學習建構其所受託代理的訊號。

English

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.