技能自我对弈：通过协同进化技能推动大语言模型能力前沿

摘要

大语言模型训练正从人工设计与标注转向基于交互驱动的自我进化。然而，现有自我进化方法在任务多样性与验证可靠性之间面临根本性困境：受限于环境的方案能获得精确反馈，却将学习局限于狭窄领域；而开放式自我生成虽拓展了任务空间，却因缺乏可靠验证，使误导性奖励污染训练循环。我们提出将智能体技能作为调和这一矛盾的有力中间地带：每种技能在特定场景中确保深度且可验证的执行，而跨技能的动态路由则维持开放式任务多样性。基于这一洞见，我们提出技能自博弈（Skill Self-Play, Skill-SP）——一个包含提案器、求解器和动态技能控制器的共进化框架。在强化学习循环的驱动下，这些组件通过持续的自博弈过程协同进化：提案器根据动态采样的技能生成具有挑战性的任务；求解器探索候选方案以突破自身能力边界；技能控制器收集执行反馈来更新和扩展技能库。这种交互式共进化有效弥合了结构化验证与开放式探索之间的鸿沟。在工具使用与推理基准上的实验评估表明，作为稳健的进化引擎，Skill-SP能持续推动强基座模型的性能天花板，同时为初始对齐不良的模型带来显著逆袭。我们的代码已开源：https://github.com/Qwen-Applications/skill-self-play。

English

LLM training is shifting from manual design and annotation to interaction-driven self-evolution. However, existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability: environment-bound methods obtain precise feedback but confine learning to narrow domains, while open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop. We identify agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety. Leveraging this insight, we introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a dynamic skill controller. Orchestrated via a reinforcement learning loop, these components co-evolve in a continuous self-play loop: the proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library. This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration. Empirical evaluations on tool-use and reasoning benchmarks demonstrate that Skill-SP, serving as a robust evolution engine, consistently pushes the performance ceiling of competent backbones while catalyzing striking turnarounds for initially misaligned models. Our code is available at https://github.com/Qwen-Applications/skill-self-play.