步驟感知偏好優化：在每個步驟上將偏好與去噪效能對齊

Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step

June 6, 2024

作者: Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, Liang Zheng

cs.AI

摘要

最近，直接偏好優化（DPO）已將其成功從對齊大型語言模型（LLMs）擴展到將文本到圖像擴散模型與人類偏好對齊。與大多數現有的DPO方法不同，這些方法假設所有擴散步驟與最終生成的圖像共享一致的偏好順序，我們認為這種假設忽略了每個步驟特定的去噪性能，並且應該為每個步驟的貢獻量定制偏好標籤。為了解決這一限制，我們提出了一種新的後訓練方法，即步驟感知偏好優化（SPO），該方法獨立評估並調整每個步驟的去噪性能，使用步驟感知偏好模型和逐步重採樣器來確保準確的步驟感知監督。具體來說，在每個去噪步驟中，我們對一組圖像進行抽樣，找到一對適當的勝負組合，更重要的是，從該組圖像中隨機選擇一個圖像來初始化下一個去噪步驟。這種逐步重採樣器過程確保下一個勝負圖像對來自同一個圖像，使勝負比較與上一步無關。為了評估每個步驟的偏好，我們訓練了一個獨立的步驟感知偏好模型，該模型可應用於噪聲和乾淨的圖像。我們使用Stable Diffusion v1.5和SDXL進行的實驗表明，SPO在對齊生成的圖像與複雜、詳細提示以及增強美學方面明顯優於最新的Diffusion-DPO，同時在訓練效率上實現了超過20倍的提升。代碼和模型：https://rockeycoss.github.io/spo.github.io/

English

Recently, Direct Preference Optimization (DPO) has extended its success from aligning large language models (LLMs) to aligning text-to-image diffusion models with human preferences. Unlike most existing DPO methods that assume all diffusion steps share a consistent preference order with the final generated images, we argue that this assumption neglects step-specific denoising performance and that preference labels should be tailored to each step's contribution. To address this limitation, we propose Step-aware Preference Optimization (SPO), a novel post-training approach that independently evaluates and adjusts the denoising performance at each step, using a step-aware preference model and a step-wise resampler to ensure accurate step-aware supervision. Specifically, at each denoising step, we sample a pool of images, find a suitable win-lose pair, and, most importantly, randomly select a single image from the pool to initialize the next denoising step. This step-wise resampler process ensures the next win-lose image pair comes from the same image, making the win-lose comparison independent of the previous step. To assess the preferences at each step, we train a separate step-aware preference model that can be applied to both noisy and clean images. Our experiments with Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms the latest Diffusion-DPO in aligning generated images with complex, detailed prompts and enhancing aesthetics, while also achieving more than 20x times faster in training efficiency. Code and model: https://rockeycoss.github.io/spo.github.io/