同策略优化中验证器诱导的支持重塑

摘要

我们证明，基于可验证奖励的在策略强化学习（RLVR）能够在改善当前目标的同时，使后续目标下的成功行为变得过于稀少而难以采样和强化。我们将这种现象称为验证器引发的支持集重塑，并将有效可奖励支持集定义为在固定采样预算内可达到的成功轨迹集合。在两类模型家族上，我们通过重复的验证器评分采样以及数学推理和受约束指令遵循上的双向训练（包括使用相反验证器进行顺序训练）来研究这一效应。Math-RLVR提高了平均指令遵循成功率，但在重复采样下减少了存在任何成功响应的提示数量。在Qwen3-8B-Base上的IFEval中，pass@1上升了6.5个百分点，而best@32下降了9.8个百分点，并且这种分歧在两种模型和IF基准上均出现。相反，IF-RLVR使数学回答从逐步解答的开头转向直接答案，在各类采样预算下降低了best@k，并减小了后续Math-RLVR的奖励变异性。词元分布分析和受控的开头干预实验表明，这些变化集中在回答的前几个词元上。RLVR主要对基策略中已有的开头进行重新排序，而所选开头对数学可搜索性具有因果影响。所测试的参考策略约束、路由先验和在策略蒸馏仅部分保留了跨任务支持集；MathIF和ReasonIF表明，边际收益仅部分转化为既正确又遵循约束的响应。因此，在在策略优化下，最终指标的提升并不能保证未来的可训练性或联合能力。代码可在 https://github.com/sylvain-wei/verifier-induced-support-reshaping 获取。

English

We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/verifier-induced-support-reshaping