技能自我對弈：以協同進化技能推展LLM能力前沿

摘要

大語言模型的訓練正從人工設計與標註，轉向以互動驅動的自我演化。然而，現有的自我演化方法在任務多樣性與驗證可靠性之間面臨根本性困境：受限於環境的方法能獲得精確回饋，但僅限於狹窄領域的學習；而開放的自我生成雖擴展了任務空間，卻缺乏可靠驗證，導致誤導性獎勵污染訓練循環。我們發現，智能體技能是調和此矛盾的強大中間地帶：每個技能確保特定情境下的深度、可驗證執行，而技能間的動態路由則維持開放的任務多樣性。基於此洞察，我們提出技能自我對弈（Skill Self-Play, Skill-SP），這是一個包含提案者、求解者與動態技能控制器的共同演化框架。透過強化學習循環的協調，這些組件在連續的自我對弈循環中共同演化：提案者根據動態取樣的技能產生具挑戰性的任務；求解者探索候選解決方案以突破能力邊界；技能控制器則收集執行回饋，以更新並擴充技能庫。這種互動式共同演化有效彌合了結構化驗證與開放式探索之間的鴻溝。在工具使用與推理基準上的實證評估顯示，Skill-SP 作為一個穩健的演化引擎，持續推升強力基線模型的能力上限，同時對初始不匹配的模型促成驚人的逆轉突破。我們的程式碼已開源於 https://github.com/Qwen-Applications/skill-self-play。

English

LLM training is shifting from manual design and annotation to interaction-driven self-evolution. However, existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability: environment-bound methods obtain precise feedback but confine learning to narrow domains, while open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop. We identify agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety. Leveraging this insight, we introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a dynamic skill controller. Orchestrated via a reinforcement learning loop, these components co-evolve in a continuous self-play loop: the proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library. This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration. Empirical evaluations on tool-use and reasoning benchmarks demonstrate that Skill-SP, serving as a robust evolution engine, consistently pushes the performance ceiling of competent backbones while catalyzing striking turnarounds for initially misaligned models. Our code is available at https://github.com/Qwen-Applications/skill-self-play.