在策略優化中的驗證器誘發支撐重塑

摘要

我們證明，採用可驗證獎勵的在策略強化學習（RLVR）在改善當前目標的同時，可能使後續目標的成功行為過於罕見，以致無法被採樣並加以強化。我們將此稱為驗證器誘發的支撐集重塑，並將有效可獎勵支撐集定義為在固定 rollout 預算內可達到的成功軌跡。在兩個模型系列中，我們透過重複的驗證器評分採樣，以及在數學推理和受限指令跟隨上的雙向訓練（包括使用相反驗證器的順序訓練）來研究此效應。數學 RLVR 提高了平均指令跟隨成功率，但在重複採樣下，減少了至少有一個成功回應的提示數量。在 Qwen3-8B-Base 於 IFEval 上的結果中，pass@1 上升了 6.5 個百分點，而 best@32 下降了 9.8 個百分點；同樣的分歧也出現在兩個模型和 IF 基準上。相反地，IF-RLVR 使數學回應從逐步推導的開頭轉向直接答案，降低了各採樣預算下的 best@k，並減少了後續 Math-RLVR 的獎勵變異。Token 分佈分析和受控的開頭干預實驗表明，這些變化集中在回應的前幾個 token 上。RLVR 主要對基礎策略中已有的開頭進行重新排序，而所選的開頭會因果性地影響數學的可搜尋性。所測試的參考策略約束、路由先驗和在策略蒸餾只能部分保留跨任務支撐集；MathIF 和 ReasonIF 表明，邊際收益僅部分轉化為既正確又遵循約束的回應。因此，端點改進並不能保證在策略最佳化下的未來可訓練性或聯合能力。程式碼可在 https://github.com/sylvain-wei/verifier-induced-support-reshaping 取得。

English

We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/verifier-induced-support-reshaping