框架更新并非框架收益：厘清自进化大语言模型智能体的进化能力

摘要

大型语言模型代理越来越多地被部署为围绕可编辑外部框架构建的系统，这些框架包括提示、技能、记忆和工具，在不改变模型参数的情况下塑造任务执行过程。框架自我进化通过从执行经验中更新这些框架来适配此类代理。然而目前尚不清楚模型在任务解决方面的基础能力是否能够预测其在框架自我进化方面的能力：哪些模型能产生有用的框架更新，哪些模型又能真正从中受益？我们分析了两类框架自我进化能力：（i）框架更新能力，即从执行经验中产生有用的持久性框架更新的能力；（ii）框架收益能力，即在任务解决过程中受益于更新后框架的能力。我们的分析揭示了两项发现。第一，框架更新能力在基础能力上呈现平缓趋势：不同能力层级的模型产生的框架更新所导致的性能提升惊人地相似；甚至Qwen3.5-9B的更新带来的收益也接近Claude Opus~4.6的水平。第二，框架收益能力在基础能力上呈现非单调性：弱层级模型从更新后的框架中受益甚微，中层模型受益最大，而强层级模型的受益程度低于中层。我们将弱层级的低收益归因于两种失败模式：弱层级模型可能无法激活相关的框架构件，或者虽然激活了构件但未能忠实地遵循其指引。这些发现表明应将能力预算投入到任务解决代理而非进化器上，并在代理训练中聚焦于框架调用和长期指令遵循能力。我们的源代码已在https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution公开。

English

LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.