DPOK：强化学习用于微调文本到图像扩散模型

摘要

研究表明，从人类反馈中学习可以改善文本到图像模型。这些技术首先学习捕捉人类在任务中关心的奖励函数，然后基于学习的奖励函数改进模型。尽管已经研究了相对简单的方法（例如，基于奖励分数的拒绝抽样），但利用奖励函数对文本到图像模型进行微调仍然具有挑战性。在这项工作中，我们提出使用在线强化学习（RL）来微调文本到图像模型。我们专注于扩散模型，将微调任务定义为一个RL问题，并使用策略梯度来更新预训练的文本到图像扩散模型，以最大化经过反馈训练的奖励。我们的方法，命名为DPOK，将策略优化与KL正则化相结合。我们对RL微调和监督微调的KL正则化进行了分析。在我们的实验中，我们展示了DPOK在图像文本对齐和图像质量方面通常优于监督微调。

English

Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an RL problem, and updating the pre-trained text-to-image diffusion models using policy gradient to maximize the feedback-trained reward. Our approach, coined DPOK, integrates policy optimization with KL regularization. We conduct an analysis of KL regularization for both RL fine-tuning and supervised fine-tuning. In our experiments, we show that DPOK is generally superior to supervised fine-tuning with respect to both image-text alignment and image quality.

DPOK：强化学习用于微调文本到图像扩散模型

DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models

摘要

Support