扩展规模的强化学习用于扩散模型

摘要

文本到图像扩散模型是一类深度生成模型，展现出了出色的高质量图像生成能力。然而，这些模型容易受到来自网络规模文本-图像训练对的隐性偏见的影响，可能无法准确地对我们关心的图像方面进行建模。这可能导致次优样本、模型偏见以及与人类伦理和偏好不符的图像。本文提出了一种有效的可扩展算法，利用强化学习（RL）跨多种奖励函数（如人类偏好、组合性和公平性）改进扩散模型，涵盖了数百万张图像。我们阐述了我们的方法如何显著优于现有方法，使扩散模型与人类偏好保持一致。我们进一步阐明了如何显著改进预训练的稳定扩散（SD）模型，生成的样本被人类偏好的比例为80.3%，优于基础SD模型的样本，同时改善了生成样本的组成和多样性。

English

Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise from web-scale text-image training pairs and may inaccurately model aspects of images we care about. This can result in suboptimal samples, model bias, and images that do not align with human ethics and preferences. In this paper, we present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL) across a diverse set of reward functions, such as human preference, compositionality, and fairness over millions of images. We illustrate how our approach substantially outperforms existing methods for aligning diffusion models with human preferences. We further illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time over those from the base SD model while simultaneously improving both the composition and diversity of generated samples.

扩展规模的强化学习用于扩散模型

Large-scale Reinforcement Learning for Diffusion Models

摘要

Support