FlowBalance：以驗證器為基礎、源自於在策略推理經驗的自我改進

摘要

推理模型能從自身的同策略經驗中改進，但此內部迴圈相當脆弱：終端驗證器提供可靠但稀疏的監督，而密集的模型自身引導則可能強化虛假信心，或使學習過度集中於狹窄的解題模式。我們提出 FlowBalance，一種基於驗證器的自我改進方法，在完整回應上學習正規化分佈。對每一條同策略軌跡，同一政策在訓練時的凍結視角利用特權上下文產生 token 層級的對數機率增益，並聚合為軌跡層級的自我引導分數。FlowBalance 以驗證器推導的群組優勢校準此分數：引導在正優勢軌跡上予以保留，在負優勢軌跡上予以反向，而當 rollout 群組未顯示結果偏好時則予以停用。由此產生的能量以指數形式對參考策略重新加權；剖析式軌跡平衡則以每個 rollout 群組一次對數配分估計來擬合正規化目標。此方法透過軌跡平衡實現結果校準式自我引導，無需額外的 token 層級模仿損失。我們的分析建立了群組內對比保持、最小變化的反向 KL 散度刻畫、驗證器對目標獎勵的單調控制，以及針對被拒絕回應上假陽性自我引導的精確校正。在數學推理方面，FlowBalance 在 Qwen3-4B 與 Qwen3-8B 上均較 FlowRL 提升平均表現，同時提升訓練速度與穩定性，避免直接 OPSD 的回覆長度崩塌，並在受控的 AIME24 診斷實驗中展現更高的正確策略多樣性。

English

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.