HarnessDev：大型語言模型能否創建並演化自己的智能體框架？

摘要

隨著代理（agent）從研究原型走向實際部署的工具，其能力日益取決於模型外部的執行基礎設施，也就是一般所稱的代理執行框架（agent harness）。在固定模型權重不變的情況下更動此執行框架，即可能大幅改變任務效能。目前的代理評測通常是在所選定的執行框架下報告下游效能；相對而言，模型自身開發執行框架的能力，仍較少被探討。我們提出 HarnessDev，這是一個將評測單位從任務輸出轉移至可執行基礎設施的基準。HarnessDev 涵蓋兩個階段：在「建立」（Creation）階段，代理從最小的種子與少數案例出發，建構出完整的執行系統；在「演化」（Evolution）階段，代理則從自身建立的執行框架出發，運用下游執行回饋反覆修訂，目標是提升基準效能。接著，我們以能力（在保留（held-out）基準上的任務成功率）與效率（執行 token 成本）兩個面向，評測每個建構出的執行框架。本文報告的建立階段結果涵蓋六個建立者 LLM、四個領域、五個下游基準，共計 2,207 個不重複的下游實例；其中隱藏的評測任務刻意不納入開發過程。我們發現，所生成的執行框架在程式碼以及搜尋與研究方面仍明顯落後於成熟的人工工程參考框架，但在寫作與機器學習實驗方面則達到或超越所選參考框架，且執行成本差異極大。演化階段能帶來若干效能增益，但這些增益並不穩定，且僅能部分遷移至保留任務。以固定執行模型進行實驗的結果進一步顯示，這些增益高度取決於實際執行該框架的模型，反映出跨模型的遷移能力有限。

English

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.