入口紧锁，内部开放：可验证奖励强化学习在何处收窄解空间

摘要

基于可验证奖励的强化学习（RLVR）能显著提升单次采样准确率（pass@1），但会导致策略的解空间收缩，从而削弱测试时扩展的收益。在本工作中，我们研究了这种广度在推理轨迹内部何处丢失：是策略无法访问某个有效解族，还是策略在计算启动后无法执行？为了将访问失败与执行失败区分开，我们分析了 Countdown 任务——其解空间可依据第一个操作数和运算符被穷举划分为离散的入口解族——并在 Qwen2.5-3B 上的 PPO 与 Qwen2.5-3B-Instruct 上的 GRPO 两种设置下进行了实验。在两种训练设置下，解覆盖度最多下降 67%，即使是在所有检查点均被解决的问题上，覆盖度也减半。我们发现，这种收缩高度集中在入口处：第一次算术运算之前的逐词元似然偏移量比后续推理过程中的偏移量大 11 至 16 倍。仅提供一个未被选中的入口前缀，就能使低访问率入口解族的完成率恢复一个数量级以上（PPO 下从 0.018 提升到 0.212），这表明替代解仍然可以被执行，只是不再被启动。在该定位结果的指导下，我们发现表层提示无法恢复多样性，而针对入口的干预是有效的：用早期检查点对后层参数进行插值，可在 pass@1 无损失的情况下将解覆盖度提高 37%。最后，我们表明，早期步骤的熵坍缩在使用 7B 和 14B 模型的六个数学基准上反复出现，但这并非推理优化的必然副产品：SFT 基线保留的覆盖度是 RLVR 方案的两倍以上，且分阶段的 SFT-DPO-RLVR 流水线能够保留早期步骤的熵。综上所述，推理广度丢失在入口处，而非房间内部。代码：https://github.com/ershiyidian/early-branch-locking。

English

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.