通过步骤级优势选择稳定高效推理

摘要

大型语言模型（LLM）通过在推理阶段分配大量计算资源，生成冗长的推理轨迹来实现强劲的推理性能。尽管近期高效推理研究通过基于长度的奖励或剪枝来降低开销，但许多方法在后训练时使用的上下文窗口远小于基座模型训练长度，这一因素的影响尚未被系统性地分离研究。我们首先证明，仅采用标准GRPO（无任何长度感知目标）进行短上下文后训练，虽能实现显著的推理压缩，但会伴随训练动态日益不稳定和准确率下降的问题。为此，我们提出步骤级优势选择（SAS）方法，该方法在推理步骤层面运作：对正确推演中的低置信度步骤与验证失败推演中的高置信度步骤均赋予零优势值——此类失败往往源于截断或验证器问题而非推理错误。在多样化数学与通用推理基准测试中，SAS相较最强长度感知基线将平均Pass@1准确率提升0.86个百分点，同时将平均推理长度降低16.3%，实现了更优的准确率-效率权衡。

English

Large language models (LLMs) achieve strong reasoning performance by allocating substantial computation at inference time, often generating long and verbose reasoning traces. While recent work on efficient reasoning reduces this overhead through length-based rewards or pruning, many approaches are post-trained under a much shorter context window than base-model training, a factor whose effect has not been systematically isolated. We first show that short-context post-training alone, using standard GRPO without any length-aware objective, already induces substantial reasoning compression-but at the cost of increasingly unstable training dynamics and accuracy degradation. To address this, we propose Step-level Advantage Selection (SAS), which operates at the reasoning-step level and assigns a zero advantage to low-confidence steps in correct rollouts and to high-confidence steps in verifier-failed rollouts, where failures often arise from truncation or verifier issues rather than incorrect reasoning. Across diverse mathematical and general reasoning benchmarks, SAS improves average Pass@1 accuracy by 0.86 points over the strongest length-aware baseline while reducing average reasoning length by 16.3%, yielding a better accuracy-efficiency trade-off.