Task-CoEvolve：透過自適應驗證任務選擇實現高效測試工具優化

摘要

我們提出了一種通過自適應驗證任務選擇來實現高效 LLM 評估框架優化的新方法。評估框架優化會根據驗證性能反覆改寫評估框架代碼，從而在不更新底層模型權重的情況下實現顯著的性能提升。然而，現有方法在每次迭代中都會完整評估一個固定的驗證集，即使在評估框架演進過程中某些任務的區分度降低，仍會產生大量評估成本。我們提出了 Task-CoEvolve，它通過解決兩個挑戰來使驗證任務與評估框架共同演進：選擇具有信息量的任務，以及從部分評估中估計完整集合的性能。Task-CoEvolve 基於以下觀察：候選評估框架之間存在分歧的任務，比起那些一直被解決或失敗的任務，更能有效區分這些評估框架。它根據過去的結果使用方差加權抽樣，將評估重點放在能力前沿附近的任務上，並且抽樣分佈會隨著評估框架的演進而自適應調整。然後，它通過考慮抽樣概率，從抽樣任務中估計完整集合的分數，從而在評估不同子集的情況下，仍能實現跨迭代的一致比較。在線上文本分類和 Terminal-Bench 2.1 上的實驗表明，Task-CoEvolve 持續優於基於子集的基線方法，並且在將優化過程中的評估次數減少 80% 的同時，達到了與完整集合搜索相當的最終性能。代碼將發佈於 https://github.com/Agent4Science-UTokyo/Task-CoEvolve。

English

We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose Task-CoEvolve, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.