SkillRise：面向跨任务技能演化的智能体强化学习

摘要

大型语言模型代理常常会遇到相互关联但各不相同的任务，这些任务共享可复用的解决方案模式。然而，标准的代理强化学习将每个任务视为独立回合，而现有的技能学习方法要么聚焦于同一任务的重复尝试，要么采用包含多个阶段的流水线，将技能提取、检索与执行纠缠在一起。我们提出SkillRise，一种统一的强化学习框架，用于跨任务学习技能。SkillRise将相关实例组织成难度递增的序列，并使用单一策略在任务求解与整理一份不断演化的技能文档之间交替，该文档直接传递给下一个任务。跨任务的解耦信用分配机制，利用当前任务结果监督求解过程，并利用折现的后续任务结果监督技能整理过程。在ALFWorld、WebShop和ScienceWorld上的实验表明，SkillRise在所有比较方法中实现了最强的Pass@1性能，相比最强基线提升幅度在2.3到8.5个百分点之间。尽管是在不同任务上进行训练，其学到的整理策略在同一任务的重复尝试中仍然有效。进一步分析揭示了测试时跨任务的扩展规律：即使每个任务只尝试一次，随着相关任务序列的加长，性能也会提升。这一趋势表明，SkillRise复用跨任务的可迁移技能，而非依赖同一任务的重复采样。与此同时，SkillRise在显著降低多阶段技能学习流水线运行时开销的情况下，仍能保持强劲性能。综合来看，这些结果为大型语言模型代理提供了一种简单高效的训练范式，使其能够跨任务提取、精炼并复用可迁移技能。

English

Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related instances into progressively challenging sequences and uses a single policy to alternate between task solving and curating an evolving skill document passed directly to the next task. Decoupled credit assignment across tasks supervises solving with the current task outcome and curation with discounted downstream outcomes. Experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the strongest Pass@1 performance among the compared methods, with gains over the strongest baseline ranging from 2.3 to 8.5 percentage points. Although trained across distinct tasks, its learned curation policy remains effective for repeated attempts on the same task. Further analysis reveals scaling at test time across tasks: performance improves with longer sequences of related tasks even when each task is attempted only once. This trend suggests that SkillRise reuses transferable skills across tasks rather than benefiting from repeated sampling of the same task. SkillRise further retains strong performance while substantially reducing the runtime overhead of skill learning pipelines with multiple stages. Together, these results provide a simple and efficient training paradigm for LLM agents to extract, refine, and reuse transferable skills across tasks.