DiffusionART：高效、穩定且可擴展的擴散模型強化學習

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

October 8, 2026

作者: Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You

cs.AI

摘要

線上強化學習（RL）以黑箱獎勵微調擴散模型，但 Flow-GRPO 等反向過程方法因信用分配不佳而效率受限；而 DiffusionNFT 等正向過程方法雖高效，卻在未仔細調整指數移動平均（EMA）離策略訓練時不穩定。我們表明，NFT 式目標含有一種隱藏的正向—反向一致性作用力，聯合正—負訓練只能部分抵消它；而 EMA 是透過回復而非離策略 rollout 來穩定訓練。我們提出 DiffusionART（Diffusion Reinforcement with Aggregation-First Reward Tilting，聚合優先獎勵傾斜的擴散強化），它將每個提示詞的獎勵聚合為反向過程狀態共用之獎勵提升方向，從構造上移除該一致性作用力，並加入受信任區域遮罩的第二更新，以實現漂移自適應回復，且無需調整 EMA。它不需要可微分獎勵模型，在每個提示詞樣本較少時仍有效，並支援成對獎勵模型（DiffusionART-Pairwise）。在圖像與影片生成中，DiffusionART 無需逐任務調參即可高效且穩定地訓練。尤其，在 HunyuanVideo-13B 上，以同等權重的視覺與運動品質獎勵，DiffusionART 所需 GPU 時數分別比 DiffusionNFT 與 Flow-GRPO-Fast 少 12.02 倍與 11.72 倍，即達到其後期平均獎勵水準，同時平均訓練獎勵分別高出 28.8% 與 53.0%。

English

Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.