弱到強擴散與反射

Weak-to-Strong Diffusion with Reflection

February 1, 2025

作者: Lichen Bai, Masashi Sugiyama, Zeke Xie

cs.AI

摘要

擴散生成模型的目標是通過梯度分數匹配來對齊學習到的分佈與真實數據分佈。然而，訓練數據質量、建模策略和架構設計中固有的限制導致生成輸出與真實數據之間存在必然差距。為了減少這種差距，我們提出了弱到強擴散（W2SD）的新框架，該框架利用現有弱模型和強模型之間的估計差異（即弱到強差異）來近似理想模型與強模型之間的差距。通過採用交替進行去噪和反演的反射操作，我們從理論上理解到，W2SD將潛在變量沿著採樣軌跡引導至真實數據分佈的區域。W2SD具有高度靈活性和廣泛應用性，通過策略性地選擇弱到強模型對（例如，DreamShaper vs. SD1.5，MoE中的優秀專家 vs. 糟糕專家），實現多樣化改進。大量實驗表明，W2SD顯著提高了人類偏好、美學質量和提示遵循，實現了各種模態（例如，圖像、視頻）、架構（例如，基於UNet、DiT、MoE）和基準的SOTA性能。例如，搭配W2SD的Juggernaut-XL可以將HPSv2勝率提高至原始結果的90%。此外，W2SD實現的性能增益明顯超過其額外的計算開銷，而來自不同弱到強差異的累積改進進一步鞏固了其實際效用性和可部署性。

English

The goal of diffusion generative models is to align the learned distribution with the real data distribution through gradient score matching. However, inherent limitations in training data quality, modeling strategies, and architectural design lead to inevitable gap between generated outputs and real data. To reduce this gap, we propose Weak-to-Strong Diffusion (W2SD), a novel framework that utilizes the estimated difference between existing weak and strong models (i.e., weak-to-strong difference) to approximate the gap between an ideal model and a strong model. By employing a reflective operation that alternates between denoising and inversion with weak-to-strong difference, we theoretically understand that W2SD steers latent variables along sampling trajectories toward regions of the real data distribution. W2SD is highly flexible and broadly applicable, enabling diverse improvements through the strategic selection of weak-to-strong model pairs (e.g., DreamShaper vs. SD1.5, good experts vs. bad experts in MoE). Extensive experiments demonstrate that W2SD significantly improves human preference, aesthetic quality, and prompt adherence, achieving SOTA performance across various modalities (e.g., image, video), architectures (e.g., UNet-based, DiT-based, MoE), and benchmarks. For example, Juggernaut-XL with W2SD can improve with the HPSv2 winning rate up to 90% over the original results. Moreover, the performance gains achieved by W2SD markedly outweigh its additional computational overhead, while the cumulative improvements from different weak-to-strong difference further solidify its practical utility and deployability.