为何推理模型会失去覆盖？——数据与分岔路口的作用

摘要

近年来，大型语言模型的进展催生了推理模型的出现，这类模型通过专门的微调流程在复杂任务上展现出强大性能。尽管这些方法可靠地提升了pass@1准确率，但先前研究观察到其存在覆盖范围收缩行为，即pass@k指标相较于基模型出现退化。本文旨在探究基于SFT的后训练中出现的推理收缩现象。我们假设这一行为源于微调数据的特性，特别是与"决策点"或"十字路口"场景相关——即当模型面临难以辨别的模式且存在多条有效推理路径时。为验证该假设，我们设计了受控案例研究模拟此类决策点场景，涵盖图分支中的不可分辨节点以及推理模式。通过追踪这些场景下的后训练动态，我们发现收缩现象与训练数据中决策点场景的普遍程度密切相关。同时，我们证明通过针对性的决策点数据合成设计，以及更具系统性的多样性激励解码机制，可在一定程度上缓解这一收缩行为。我们的研究结果将数据中心因素确定为推理模型收缩的关键驱动力，并强调多样性感知设计是控制该行为的有效杠杆。

English

Recent progress in large language models has led to the emergence of reasoning models, which have shown strong performance on complex tasks through specialized fine-tuning procedures. While these methods reliably improve pass@1 accuracy, prior works have observed that they show a coverage shrinkage behavior, where pass@k degrades relative to the base model. In this paper, we investigate the reasoning shrinkage arise under SFT-based post-training. We hypothesize that this behavior is driven by properties of the fine-tuning data, specifically related to decision points or "forks in the road" scenarios where model faces indecipherable patterns with multiple valid reasoning paths. To test this hypothesis, we design controlled case studies that simulate such decision-point settings, spanning indecipherable nodes in graph branching, and reasoning modes. By tracking post-training dynamics in these settings, we find that the shrinkage phenomenon is tightly correlated with the prevalence of decision-point scenarios in the training data. We also demonstrate that this shrinkage behavior can be partially mitigated through targeted data synthesis design of decision-points, and a more systematic diversity-encouraging decoding mechanism. Our findings identify data-centric factors as a key driver of shrinkage in reasoning models and highlight diversity-aware designs as an effective lever for controlling it.