SkillRise：跨任务技能演化的自主強化學習

摘要

大型語言模型代理經常遇到相關但不同的任務，這些任務共享可重複使用的解決方案模式。然而標準的代理強化學習將任務視為獨立的情節，而現有的技能學習方法若非專注於單一任務的重複嘗試，就是使用多階段的管道來混雜提取、檢索和執行。我們提出 SkillRise，這是一個用於跨任務學習技能的統一強化學習框架。SkillRise 將相關實例組織成漸進挑戰的序列，並使用單一策略在任務解決與策劃直接傳遞給下一個任務的動態技能文件之間交替。跨任務的解耦信用分配使用當前任務結果來監督解決，並使用折扣的後續任務結果來監督策劃。在 ALFWorld、WebShop 和 ScienceWorld 上的實驗顯示，SkillRise 在比較方法中達到了最強的 Pass@1 表現，相較於最強基線的增益範圍為 2.3 到 8.5 個百分點。儘管在不同任務上進行訓練，其學到的策劃策略在相同任務的重複嘗試中仍然有效。進一步分析顯示，在測試時跨任務存在擴展性：即使每個任務僅嘗試一次，隨著相關任務序列的增長，表現也會提升。這一趨勢表明，SkillRise 跨任務重用可遷移的技能，而非從相同任務的重複抽樣中受益。此外，SkillRise 在大幅減少多階段技能學習管道的運行時開銷的同時，仍保持強勁的表現。綜合這些結果，為 LLM 代理提供了一種簡單且高效的訓練範式，使其能夠跨任務提取、精煉並重用可遷移的技能。

English

Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related instances into progressively challenging sequences and uses a single policy to alternate between task solving and curating an evolving skill document passed directly to the next task. Decoupled credit assignment across tasks supervises solving with the current task outcome and curation with discounted downstream outcomes. Experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the strongest Pass@1 performance among the compared methods, with gains over the strongest baseline ranging from 2.3 to 8.5 percentage points. Although trained across distinct tasks, its learned curation policy remains effective for repeated attempts on the same task. Further analysis reveals scaling at test time across tasks: performance improves with longer sequences of related tasks even when each task is attempted only once. This trend suggests that SkillRise reuses transferable skills across tasks rather than benefiting from repeated sampling of the same task. SkillRise further retains strong performance while substantially reducing the runtime overhead of skill learning pipelines with multiple stages. Together, these results provide a simple and efficient training paradigm for LLM agents to extract, refine, and reuse transferable skills across tasks.