DiffusionART：高效、稳定且可扩展的扩散模型强化方法

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

October 8, 2026

作者: Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You

cs.AI

摘要

在线强化学习（RL）利用黑盒奖励微调扩散模型，但诸如 Flow-GRPO 之类的反向过程方法因信用分配不佳而效率受限，而诸如 DiffusionNFT 之类的前向过程方法虽然高效，但在没有精心调参的指数移动平均（EMA）离策略训练下并不稳定。我们表明，NFT 风格的目标函数包含一种隐藏的前向-反向一致性力，联合正负训练只能部分抵消它，而 EMA 通过恢复而非离策略 rollout 来稳定训练。我们提出 DiffusionART（Diffusion Reinforcement with Aggregation-First Reward Tilting，聚合优先奖励倾斜的扩散强化学习），它将每个提示词的奖励聚合为反向过程状态的一个共享的奖励改进方向，从构造上消除了该一致性力，并添加了信赖域掩码的第二次更新，以实现无需 EMA 调参的漂移自适应恢复。它不需要可微奖励模型，在每提示词样本更少时仍然有效，并支持成对奖励模型（DiffusionART-Pairwise）。在图像和视频生成中，DiffusionART 无需逐任务调参即可高效稳定地训练。特别是在 HunyuanVideo-13B 上，使用等权重的视觉和运动质量奖励时，DiffusionART 达到 DiffusionNFT 和 Flow-GRPO-Fast 后期平均奖励水平所需的 GPU 小时数分别减少了 12.02 倍和 11.72 倍，同时平均训练奖励分别高出 28.8% 和 53.0%。

English

Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.