HarnessDev：大语言模型能否创建并演进自身的智能体框架？

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

September 1, 2026

作者: Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, Wenxuan Zhang

cs.AI

摘要

随着代理从研究原型走向部署工具，其能力日益依赖于模型外部的执行基础设施，通常称为代理执行框架（agent harness）。在固定模型权重的情况下改变这一执行框架，会显著改变任务性能。当前的代理评测通常报告在特定执行框架下的下游性能，而模型自身开发执行框架的能力相对未被充分探索。我们提出了 HarnessDev 基准，它将评测单元从任务输出转变为可运行的执行基础设施。HarnessDev 涵盖两个阶段。在创建（Creation）阶段，代理从极简的种子代码和少量用例出发，构建完整的执行系统。在演进（Evolution）阶段，代理从自身创建的执行框架开始，利用下游执行反馈进行迭代修改，以提升基准性能为目标。随后，我们评估每个构建出的执行框架的能力（在保留基准上的任务成功率）和效率（执行令牌成本）。所报告的创建结果涵盖六个创建者 LLM、四个领域和五个下游基准，共包含 2,207 个独立的下游实例，并预留了隐藏评测任务以防止在开发过程中泄露。我们发现，生成的执行框架在代码、搜索与研究领域仍明显落后于成熟的人工设计参考，而在写作和机器学习实验领域则与所选参考相当甚至更优，同时执行成本差异很大。演进产生了一定的性能提升，但效果不稳定，且只能部分迁移到保留任务上。使用固定运行时模型的实验进一步表明，这些收益在很大程度上依赖于执行该框架的模型，说明跨模型的迁移能力有限。

English

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.