你所看到的就是你所阅读的吗？改进文本-图像对齐评估

摘要

自动确定文本和相应图像是否语义对齐是视觉语言模型面临的重要挑战，具有生成文本到图像和图像到文本任务的应用。在这项工作中，我们研究了用于自动文本-图像对齐评估的方法。我们首先介绍了SeeTRUE：一个全面的评估集，涵盖了来自文本到图像和图像到文本生成任务的多个数据集，其中包含人类判断给定文本-图像对是否语义对齐的数据。然后，我们描述了两种自动确定对齐的方法：第一种方法涉及基于问题生成和视觉问题回答模型的流程，第二种方法采用端到端分类方法，通过微调多模态预训练模型。这两种方法在各种文本-图像对齐任务中均优于先前的方法，在涉及复杂构图或不自然图像的挑战性案例中取得了显著改进。最后，我们演示了我们的方法如何定位图像和给定文本之间的特定不对齐，并展示了它们如何用于自动重新排列文本到图像生成中的候选项。

English

Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

你所看到的就是你所阅读的吗？改进文本-图像对齐评估

What You See is What You Read? Improving Text-Image Alignment Evaluation

摘要

Support