HarnessOpt-Bench：評估大型語言模型於測試框架最佳化之表現

摘要

隨著大型語言模型（LLM）日益部署於智能體系統中，其能力不僅取決於模型權重，還取決於框架（harness）：即圍繞模型的提示詞、工具、控制流程、記憶體與編排程式碼。這使得自動化框架優化——由人工智慧系統對框架進行迭代式、以評估為導向的改進——既是提升人工智慧系統的重要途徑，也是人工智慧系統本身所需的高要求能力。然而，學界目前缺乏一套通用協議來衡量前沿大型語言模型在此任務上的表現。我們引入 HarnessOpt-Bench，一個針對昂貴且隨機評估情境下的端到端框架優化基準。優化器——一個配對編碼框架的大型語言模型——接收目標智能體的種子框架、分級評估回饋，以及固定的目標評估預算。它編輯框架並提名最終候選方案，該方案以相對於種子在保留測試分割上的歸一化增益進行評分，而該測試分割在整個搜尋過程中均無法存取。可信執行環境強制執行評估邊界、計量目標智能體的資源使用情況，並保留候選版本以供稽核。我們在共享編碼框架及其原生框架下，將 5 個前沿大型語言模型作為優化器，在 4 個下游任務中進行評估，共進行 111 次評分運行。實驗結果顯示，優化器模型之間的差異大於其所藉以運作的編碼框架之間的差異，原生框架並非一致地優越，且增益在不同任務與種子機制間存在顯著差異。這些結果確立了框架優化作為一項可衡量且具區辨力的能力，且尚有大幅改進空間。

English

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.