文本到图像生成的丰富人类反馈

摘要

最近的文本到图像（T2I）生成模型，如稳定扩散和Imagen，在基于文本描述生成高分辨率图像方面取得了显著进展。然而，许多生成的图像仍然存在问题，如瑕疵/不合理性、与文本描述不一致以及美学质量低下。受强化学习与人类反馈（RLHF）在大型语言模型中取得成功的启发，先前的研究收集了人类提供的分数作为对生成图像的反馈，并训练了一个奖励模型来改善T2I生成。在本文中，我们通过（i）标记图像中不合理或与文本不一致的区域，以及（ii）注释文本提示中被错误呈现或缺失在图像上的单词，丰富了反馈信号。我们在1.8万个生成的图像上收集了这样丰富的人类反馈，并训练了一个多模态变压器来自动预测丰富的反馈。我们展示了预测的丰富人类反馈可以用来改善图像生成，例如，通过选择高质量的训练数据来微调和改进生成模型，或者通过使用预测的热图创建掩模来修复问题区域。值得注意的是，这些改进可以推广到超出用于生成人类反馈数据的图像的模型（Muse）上，这些模型超出了用于收集数据的模型（稳定扩散变体）。

English

Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. Inspired by the success of Reinforcement Learning with Human Feedback (RLHF) for large language models, prior works collected human-provided scores as feedback on generated images and trained a reward model to improve the T2I generation. In this paper, we enrich the feedback signal by (i) marking image regions that are implausible or misaligned with the text, and (ii) annotating which words in the text prompt are misrepresented or missing on the image. We collect such rich human feedback on 18K generated images and train a multimodal transformer to predict the rich feedback automatically. We show that the predicted rich human feedback can be leveraged to improve image generation, for example, by selecting high-quality training data to finetune and improve the generative models, or by creating masks with predicted heatmaps to inpaint the problematic regions. Notably, the improvements generalize to models (Muse) beyond those used to generate the images on which human feedback data were collected (Stable Diffusion variants).

文本到图像生成的丰富人类反馈

Rich Human Feedback for Text-to-Image Generation

摘要

Support