通过优化人类效用来对齐扩散模型

Aligning Diffusion Models by Optimizing Human Utility

April 6, 2024

作者: Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka

cs.AI

摘要

我们提出了Diffusion-KTO，这是一种新颖的方法，用于通过将对齐目标定义为最大化期望人类效用来对齐文本到图像扩散模型。由于这一目标适用于每一代独立地，Diffusion-KTO不需要收集昂贵的成对偏好数据，也不需要训练复杂的奖励模型。相反，我们的目标需要简单的每个图像的二进制反馈信号，例如喜欢或不喜欢，这些信号是丰富可获得的。经过Diffusion-KTO的微调后，文本到图像扩散模型在人类判断和自动评估指标（如PickScore和ImageReward）方面表现优越，超过了现有技术，包括监督微调和Diffusion-DPO。总的来说，Diffusion-KTO释放了利用易获得的每个图像二进制信号的潜力，并扩大了将文本到图像扩散模型与人类偏好对齐的适用性。

English

We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.