HarnessOpt-Bench：评估大型语言模型在流程优化中的表现

摘要

随着大语言模型（LLM）越来越多地部署在智能体系统中，它们的能力不仅取决于模型权重，还取决于其外围的 harness——即围绕模型的提示、工具、控制流、记忆和编排代码。这使得自动化 harness 优化——由 AI 系统对 harness 进行迭代式、以评估为指引的改进——既是提升 AI 系统的重要途径，也是对 AI 系统自身的一项严苛能力要求。然而，社区目前缺乏一种通用协议来衡量前沿 LLM 在此任务上的表现。我们提出了 HarnessOpt-Bench，一个用于在昂贵且随机的评估条件下进行端到端 harness 优化的基准测试。优化器（一个 LLM 与一个编码 harness 配对）接收目标智能体的种子 harness、带评分的评估反馈以及固定的目标评估预算。它编辑 harness 并提名一个最终候选版本；该候选版本根据其在种子版本之上的归一化增益进行评分，评分基于在整个搜索过程中保持不可访问的预留测试分区。可信执行环境强制执行评估边界，计量目标智能体的资源使用情况，并保留各候选版本以供审计。我们在 4 个下游任务上，分别在共享编码 harness 和各自原生 harness 下，对 5 个前沿 LLM 作为优化器的表现进行了评估，共完成 111 次计分运行。实验结果表明，优化器模型之间的区分度大于其所借助的编码 harness 之间的区分度；原生 harness 并不始终更优；收益在不同任务和种子机制之间差异显著。这些结果将 harness 优化确立为一种可测量、可区分且仍有巨大提升空间的能力。

English

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.