擴展式強化學習用於擴散模型

摘要

文本到圖像擴散模型是一類深度生成模型，展示了出色的高質量圖像生成能力。然而，這些模型容易受到隱含偏見的影響，這些偏見源於規模龐大的文本-圖像訓練對，可能無法準確地建模我們關心的圖像方面。這可能導致次優質樣本、模型偏見以及與人類倫理和偏好不符的圖像。本文提出了一種有效的可擴展算法，使用強化學習（RL）跨越各種不同的獎勵函數來改進擴散模型，例如人類偏好、組成性和公平性，涵蓋數百萬圖像。我們說明了我們的方法如何顯著優於現有的方法，使擴散模型與人類偏好保持一致。我們進一步說明了這如何顯著改進了預訓練的穩定擴散（SD）模型，生成的樣本在80.3%的時間內被人類偏好，優於基本SD模型的樣本，同時改善了生成樣本的組成和多樣性。

English

Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise from web-scale text-image training pairs and may inaccurately model aspects of images we care about. This can result in suboptimal samples, model bias, and images that do not align with human ethics and preferences. In this paper, we present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL) across a diverse set of reward functions, such as human preference, compositionality, and fairness over millions of images. We illustrate how our approach substantially outperforms existing methods for aligning diffusion models with human preferences. We further illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time over those from the base SD model while simultaneously improving both the composition and diversity of generated samples.

擴展式強化學習用於擴散模型

Large-scale Reinforcement Learning for Diffusion Models

摘要

Support