基於強化學習的漸進式智能體技能生成

摘要

現有的技能生成方法在很大程度上依賴於啟發式規則或管線式整合，這些方法必須針對不同的證據來源進行特殊設計。相比之下，基於學習的方法提供了一種更統一的方式來對跨異質來源的技能生成進行建模。然而，基於學習的技能生成仍然具有挑戰性，因為技能缺乏基於相關性或正確性的自然監督信號；技能的價值在很大程度上只能通過其是否改善智能體在下游任務中的行為來判斷。為了解決這一挑戰，我們提出了Skill-α，一種用於逐步生成高質量智能體技能的強化學習方法。具體來說，我們將技能生成形式化為一個序列編輯過程，將技能構建分解為可獨立評估的編輯步驟，並引入了一種新穎的回滾獎勵機制，該機制通過在錨定查詢上比較原始技能與編輯後技能的下游執行結果來評估每次編輯。大量實驗表明，在文檔到技能和經驗到技能兩種設定下，Skill-α生成的技能都比基於啟發式或管線式的方法更有效。在以GPT-4o為主要工作智能體的情況下，與最強的技能生成基線相比，Skill-α在CL-Bench上的平均下游成功率提高了3.3個百分點，在tau2-bench上提高了6.7個百分點。進一步的消融實驗驗證了回滾獎勵和逐步生成的重要性。

English

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-α, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-α generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-α improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.