基于强化学习的渐进式智能体技能生成

摘要

现有的技能生成方法在很大程度上依赖于启发式方法或流水线式整合，而这些方法必须针对不同的证据来源进行专门设计。相比之下，基于学习的方法提供了一种更统一的途径，能够在异构来源之间对技能生成进行建模。然而，基于学习的技能生成仍然具有挑战性，因为技能缺乏基于相关性或正确性的自然监督信号；其价值在很大程度上只能通过它们是否改善智能体在下游任务上的行为来判断。为应对这一挑战，我们提出了Skill-α，一种用于逐步生成高质量智能体技能的强化学习方法。具体而言，我们将技能生成形式化为一个序列化编辑过程，将技能构建分解为可单独评估的编辑步骤，并引入一种新颖的回滚奖励机制，该机制通过比较在锚定查询上原始技能与编辑后技能的下游执行效果来评估每次编辑。大量实验表明，在从文档到技能和从经验到技能两种设置下，Skill-α生成的技能均优于基于启发式或流水线的方法。在主要的GPT-4o工作模型下，与最强的技能生成基线相比，Skill-α在CL-Bench上平均下游成功率提升了3.3个百分点，在tau2-bench上提升了6.7个百分点。进一步的消融实验验证了回滚奖励和渐进式生成的重要性。

English

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-α, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-α generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-α improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.