NormGuard：流匹配强化学习中的奖励保持范数约束

摘要

强化学习（RL）后训练改善了流生成模型的奖励对齐，但往往会在奖励代理未捕捉到的方面降低感知质量。我们发现了这种漂移的一个简单结构特征：在三种后训练方法（NFT、AWM、DPO）中，RL微调使得每步速度范数|v_θ|相对于参考模型膨胀了5%至15%。无分类器引导（CFG）中已研究过一种范数膨胀形式，通过在推理时将速度重新缩回到参考范数，可以缓解由此产生的伪影。然而，这种推理时修正并不能直接迁移到RL：在推理时将v_θ重新缩放以匹配|v_{ref}|既不能改善奖励，也无法修复质量退化，因为膨胀已与模型权重共同适应。此外，伴随灵敏度分析表明，速度幅值重缩放无法在批次层面携带一致的一阶奖励信号，这表明抑制范数膨胀不太可能移除一个持续携带奖励的分量。由于推理时重归一化失败，而范数抑制又不带来奖励代价，因此训练时的干预是合适的策略。综合这些发现，我们提出了\methodname，一种仅在|v_θ|超过|v_{ref}|时激活的铰链惩罚项，并以可加方式与任何速度局域基础损失组合。在两个基础模型、三种后训练方法和两个奖励代理上，\methodname持续改善了MLLM评定的图像质量和法医真实性，同时保持了奖励水平，且在少步推理下增益更大，且不能归因于早停。

English

Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm |v_θ| by 5% to 15% relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling v_θ to match |v_{ref}| at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate \methodname, a hinge penalty that activates only when |v_θ| exceeds |v_{ref}| and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, \methodname consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.