ChatPaper.aiChatPaper

通過優化人類效用來對齊擴散模型

Aligning Diffusion Models by Optimizing Human Utility

April 6, 2024
作者: Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka
cs.AI

摘要

我們提出了Diffusion-KTO,一種新穎的方法,用於對齊文本到圖像擴散模型,將對齊目標定義為最大化期望人類效用。由於該目標適用於每個生成獨立地,Diffusion-KTO不需要收集昂貴的成對偏好數據,也不需要訓練複雜的獎勵模型。相反,我們的目標需要簡單的每圖像二元反饋信號,例如喜歡或不喜歡,這些信號是豐富可得的。在使用Diffusion-KTO進行微調後,文本到圖像擴散模型在人類判斷和自動評估指標(如PickScore和ImageReward)方面表現優越,超越了現有技術,包括監督微調和Diffusion-DPO。總的來說,Diffusion-KTO發揮了利用易得的每圖像二元信號的潛力,擴大了對齊文本到圖像擴散模型與人類偏好的應用範圍。
English
We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Summary

AI-Generated Summary

PDF151December 15, 2024