迈向技能原生的LLM：用于长程推理基准测试与训练的技能熵

摘要

近期大语言模型中的长程推理要求模型在推理链内切换不同的技能，例如先进行数学推导，再利用推导结果来规划日程。我们将此类问题称为跨技能长程任务：这类任务包含多个步骤，各步骤需要不同的推理技能，并且依赖于前面的输出结果。现有基准通常单独评估各项技能，缺乏一种规范化的方法来衡量模型在技能之间切换的能力。我们从评估和训练两个方面来弥补这一空白。我们引入技能熵（Skill Entropy），用于度量从一种技能切换到另一种技能的难度；随后提出 Skill²-Bench——一个基于 558 个技能、覆盖 9 个可验证且开放式领域的跨技能长程任务基准。每个任务都被赋予一个任务级技能熵分数，并划分为三个难度等级。在 Skill²-Bench 上评估 8 个前沿模型和 4 个开源模型后，我们发现了一种技能切换差距：在技能熵更高的任务上，准确率会下降。接着，我们将技能熵从基准度量转化为训练信号，提出 Skill-Entropy RL——一种强化学习框架，其中模型不仅预测每一步的答案，还预测用于生成该答案的技能。奖励函数结合了步骤级正确性与技能熵奖励，后者用于衡量模型预测的技能序列与标准技能序列之间的一致性。在 Qwen3-4B-Instruct 和 Qwen3-1.7B 上，Skill-Entropy RL 将 Skill²-Bench 分数分别从 34.4% 提升至 68.4%、从 14.6% 提升至 40.1%，优于多个强基线方法。同一流程也可直接应用于 OpenR1-Math 等现成训练数据，表明技能熵是一种可复用的训练信号。代码见：https://github.com/Gen-Verse/Skill-Entropy-RL

English

Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks whose steps require different reasoning skills and depend on earlier outputs. Existing benchmarks often evaluate individual skills, lacking a principled way to measure how well a model switches between skills. We address this gap from both the evaluation and training sides. We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another. We then propose Skill^2-Bench, a benchmark of cross-skill long-horizon tasks built over 558 skills across 9 verifiable and open-ended domains. Each task is assigned a task-level skill-entropy score and grouped into three difficulty levels. Evaluating 8 frontier and 4 open-source models on Skill^2-Bench reveals a skill-switching gap: accuracy decreases on higher-entropy tasks. We then turn skill entropy from a benchmark scale into a training signal. We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it. The reward combines step-level correctness with a skill-entropy reward that measures the alignment between the model-predicted skill sequence and the gold skill sequence. On Qwen3-4B-Instruct and Qwen3-1.7B, Skill-Entropy RL improves the Skill^2-Bench score from 34.4% to 68.4% and from 14.6% to 40.1%, respectively, outperforming competitive baselines. The same pipeline can be applied to off-the-shelf training data such as OpenR1-Math, indicating that skill entropy is a reusable training signal. Code available at: https://github.com/Gen-Verse/Skill-Entropy-RL