NormGuard：流匹配強化學習中的獎勵保持範數約束

摘要

強化學習（RL）後訓練能改善基於流動之生成器的獎勵對齊，但常以獎勵代理無法捕捉的方式降低感知品質。我們發現此偏移的一項簡單結構性特徵：在三種後訓練方法（NFT、AWM、DPO）中，RL微調使每步速度範數|v_θ|相較於參考值膨脹了5%至15%。一種範數膨脹形式已在無分類器引導（CFG）中被研究，該方法在推理時將速度重新縮放回參考範數，可緩解由此產生的偽影。然而，此推理時修正無法直接遷移至RL：在推理時將v_θ重新縮放以匹配|v_{ref}|，既未提升獎勵，也未能修復品質退化，因為膨脹已與模型權重共同適應。此外，伴隨靈敏度分析顯示，速度量級的重新縮放在批次層級不攜帶一致的一階獎勵訊號，這表明抑制範數膨脹不太可能移除一個持續攜帶獎勵訊息的成分。由於推理時重正化失敗，而範數抑制又不帶來獎勵成本，因此訓練時干預是合適的策略。綜合這些發現，我們提出\methodname，這是一種僅在|v_θ|超過|v_{ref}|時啟動的鉸鏈懲罰，並可與任何基於速度局部的損失函數相加組合。在兩種基礎模型、三種後訓練方法及兩種獎勵代理上，\methodname一致地改善了MLLM評判的影像品質與鑑識真實性，同時保留了獎勵，且其增益在少步推理下更為顯著，並無法以早停解釋。

English

Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm |v_θ| by 5% to 15% relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling v_θ to match |v_{ref}| at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate \methodname, a hinge penalty that activates only when |v_θ| exceeds |v_{ref}| and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, \methodname consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.