Task-CoEvolve：基于自适应验证任务选择的高效测试框架优化

摘要

我们提出了一种通过自适应验证任务选择实现高效LLM评估框架优化的新方法。评估框架优化根据验证性能迭代地重写评估框架代码，无需更新底层模型权重即可获得显著的性能提升。然而，现有方法在每次迭代时都完整地评估固定的验证集，即使在评估框架演化中某些任务变得不太具有区分度，也会产生大量的评估成本。我们提出了Task-CoEvolve，通过解决两个挑战来使验证任务与评估框架共同演化：选择信息丰富的任务，以及从部分评估中估计完整集的性能。Task-CoEvolve基于这样一个观察：候选评估框架之间存在分歧的任务，比那些总是被解决或总是失败的任务，更能有效地区分候选框架。它使用基于过去结果的方差加权采样，将评估集中在能力前沿附近的任务上，并且采样分布会随着评估框架的演化而自适应调整。然后，它通过考虑采样概率来从采样的任务中估计完整集的分数，从而使不同迭代之间的比较保持一致，尽管评估的任务子集不同。在在线文本分类和Terminal-Bench 2.1上的实验表明，Task-CoEvolve始终优于基于子集的基线方法，与完整集搜索的最终性能相当，同时将优化过程中的评估次数减少了80%。代码将在https://github.com/Agent4Science-UTokyo/Task-CoEvolve发布。

English

We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose Task-CoEvolve, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.