知道何时停止：通过段级信用分配减少过度思考

摘要

推理语言模型经常过度思考：生成长串扩展行为，例如规避性表述、放弃当前思路以及自相矛盾，这些行为消耗了大量词元，却并未改善答案。我们表明，这些行为不仅仅是长度造成的结果；即使在控制回答长度后，错误轨迹中的无益自我反思比例仍高于正确轨迹。解决这一问题需要识别自我反思在哪些情境下有益、哪些情境下有害，但获取此类步骤级标注成本高昂。我们观察到，推理轨迹中的中间答案承诺可以提供一种廉价的代理信号：通过将轨迹中每个（拟）最终答案候选与标准答案进行比较，我们无需任何额外监督即可判断后续反思是否具有建设性。基于这一见解，我们提出 DASH（Drift Aware advantage SHaping，漂移感知优势塑形），根据每个推理片段是导向正确性还是偏离正确性来分配片段级信用。在竞赛级数学基准上，DASH 在过度思考普遍存在的场景下取得了最高准确率（平均准确率：59.45%，对比 Dr.GRPO 的 58.1% 与 GRPO 的 56.95%），同时减少了过度思考行为，并实现了比基线更具建设性的自我纠正。

English

Reasoning language models frequently overthink: generating extended chains of behaviors such as hedging, approach abandonment, and self contradiction that consume tokens without improving answers. We show that these behaviors are not merely a consequence of length; even when controlling for response length, incorrect traces exhibit higher rates of unproductive self-reflection than correct ones. Addressing this requires identifying where self-reflection helps vs hurts, but obtaining these step-level annotations is costly. We observe that intermediate answer commitments within reasoning traces can provide a cheap proxy: by comparing each final answer candidate in the trace to the ground truth, we can determine whether subsequent reflection is productive without any additional supervision. Building on this insight, we propose DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness. On competition-level math benchmarks, DASH achieves the highest accuracy where overthinking is prevalent (Average Accuracy: 59.45% vs. 58.1% Dr.GRPO vs. 56.95% GRPO) while reducing overthinking behaviors and achieving more productive self-correction than baselines.