當偏好分歧時：將擴散模型與少數群體感知的自適應DPO對齊

When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPO

March 21, 2025

作者: Lingfan Zhang, Chen Liu, Chengming Xu, Kai Hu, Donghao Luo, Chengjie Wang, Yanwei Fu, Yuan Yao

cs.AI

摘要

近年來，圖像生成領域取得了顯著進展，特別是在將模型與普世人類偏好對齊的微調方法方面。本文探討了偏好數據在擴散模型訓練過程中的關鍵作用，尤其是在Diffusion-DPO及其後續改進的背景下。我們研究了圖像生成中普世人類偏好的複雜性，強調了這些偏好的主觀性以及偏好數據集中少數樣本所帶來的挑戰。通過初步實驗，我們證明了少數樣本的存在及其對模型性能的不利影響。我們提出了Adaptive-DPO——一種將少數樣本感知指標納入DPO目標的新方法。該指標包括註釋者內置信度和註釋者間穩定性，能夠區分多數樣本和少數樣本。我們引入了一種Adaptive-DPO損失函數，該函數在兩個方面改進了DPO損失：增強模型對多數標籤的學習，同時減輕少數樣本的負面影響。我們的實驗表明，該方法能有效處理合成少數數據和真實世界偏好數據，為圖像生成任務中更有效的訓練方法鋪平了道路。

English

In recent years, the field of image generation has witnessed significant advancements, particularly in fine-tuning methods that align models with universal human preferences. This paper explores the critical role of preference data in the training process of diffusion models, particularly in the context of Diffusion-DPO and its subsequent adaptations. We investigate the complexities surrounding universal human preferences in image generation, highlighting the subjective nature of these preferences and the challenges posed by minority samples in preference datasets. Through pilot experiments, we demonstrate the existence of minority samples and their detrimental effects on model performance. We propose Adaptive-DPO -- a novel approach that incorporates a minority-instance-aware metric into the DPO objective. This metric, which includes intra-annotator confidence and inter-annotator stability, distinguishes between majority and minority samples. We introduce an Adaptive-DPO loss function which improves the DPO loss in two ways: enhancing the model's learning of majority labels while mitigating the negative impact of minority samples. Our experiments demonstrate that this method effectively handles both synthetic minority data and real-world preference data, paving the way for more effective training methodologies in image generation tasks.