邁向技能原生的大型語言模型：以技能熵進行長程推理的基準測試與訓練

摘要

近期大型語言模型（LLMs）中的長程推理要求模型在推理鏈中於不同技能之間切換，例如先進行數學推導，再利用結果規劃行程。我們將此類問題稱為跨技能長程任務：其步驟需要不同推理技能且依賴先前輸出的多步驟任務。現有基準常以單一技能評估，缺乏衡量模型在技能間切換能力的原理性方法。我們從評測與訓練兩方面填補此缺口。我們提出技能熵（Skill Entropy），用以衡量從一種技能切換至另一種技能的困難度。接著我們提出 Skill²-Bench，這是一個跨技能長程任務基準，涵蓋 9 個可驗證且開放式領域中的 558 種技能。每項任務都被賦予任務層級的技能熵分數，並劃分為三個難度等級。在 Skill²-Bench 上評估 8 個前沿模型與 4 個開源模型後，我們發現了技能切換差距：在熵值較高的任務上準確率會下降。接著我們將技能熵從基準量表轉化為訓練訊號。我們提出技能熵強化學習（Skill-Entropy RL），這是一個強化學習框架，模型不僅在每一步預測答案，也預測產生該答案所使用的技能。獎勵結合了步驟層級的正確性，以及衡量模型預測技能序列與黃金技能序列一致性的技能熵獎勵。在 Qwen3-4B-Instruct 與 Qwen3-1.7B 上，Skill-Entropy RL 將 Skill²-Bench 分數分別從 34.4% 提升至 68.4%、從 14.6% 提升至 40.1%，優於具競爭力的基線方法。相同的流程可應用於如 OpenR1-Math 等現成訓練資料，顯示技能熵是一種可重複使用的訓練訊號。程式碼位於：https://github.com/Gen-Verse/Skill-Entropy-RL

English

Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks whose steps require different reasoning skills and depend on earlier outputs. Existing benchmarks often evaluate individual skills, lacking a principled way to measure how well a model switches between skills. We address this gap from both the evaluation and training sides. We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another. We then propose Skill^2-Bench, a benchmark of cross-skill long-horizon tasks built over 558 skills across 9 verifiable and open-ended domains. Each task is assigned a task-level skill-entropy score and grouped into three difficulty levels. Evaluating 8 frontier and 4 open-source models on Skill^2-Bench reveals a skill-switching gap: accuracy decreases on higher-entropy tasks. We then turn skill entropy from a benchmark scale into a training signal. We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it. The reward combines step-level correctness with a skill-entropy reward that measures the alignment between the model-predicted skill sequence and the gold skill sequence. On Qwen3-4B-Instruct and Qwen3-1.7B, Skill-Entropy RL improves the Skill^2-Bench score from 34.4% to 68.4% and from 14.6% to 40.1%, respectively, outperforming competitive baselines. The same pipeline can be applied to off-the-shelf training data such as OpenR1-Math, indicating that skill entropy is a reusable training signal. Code available at: https://github.com/Gen-Verse/Skill-Entropy-RL