統一的文本到圖像生成與檢索

Unified Text-to-Image Generation and Retrieval

June 9, 2024

作者: Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, Tat-Seng Chua

cs.AI

摘要

人類如何有效且高效地獲取圖像一直是一個長期存在的問題。一個典型的解決方案是從現有數據庫中根據文本查詢進行文本到圖像檢索；然而，這種有限的數據庫通常缺乏創造力。相比之下，最近在文本到圖像生成方面取得的突破使得產生花俏且多樣化的視覺內容成為可能，但在合成知識密集型圖像方面面臨挑戰。在這項工作中，我們重新思考了文本到圖像生成和檢索之間的關係，並提出了一個統一的框架，放在多模態大型語言模型（MLLMs）的背景下。具體來說，我們首先探索了MLLMs的內在區分能力，並引入了一種生成式檢索方法，以無需訓練的方式進行檢索。隨後，我們以自回歸生成方式統一了生成和檢索，並提出了一個自主決策模塊，以選擇在生成和檢索的圖像中選擇最佳匹配的圖像作為對文本查詢的響應。此外，我們構建了一個名為TIGeR-Bench的基準，其中包括創意和知識密集領域，以標準化統一文本到圖像生成和檢索的評估。在TIGeR-Bench和兩個檢索基準，即Flickr30K和MS-COCO上的廣泛實驗結果證明了我們提出的方法的優越性和有效性。

English

How humans can efficiently and effectively acquire images has always been a perennial question. A typical solution is text-to-image retrieval from an existing database given the text query; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in text-to-image generation have made it possible to produce fancy and diverse visual content, but it faces challenges in synthesizing knowledge-intensive images. In this work, we rethink the relationship between text-to-image generation and retrieval and propose a unified framework in the context of Multimodal Large Language Models (MLLMs). Specifically, we first explore the intrinsic discriminative abilities of MLLMs and introduce a generative retrieval method to perform retrieval in a training-free manner. Subsequently, we unify generation and retrieval in an autoregressive generation way and propose an autonomous decision module to choose the best-matched one between generated and retrieved images as the response to the text query. Additionally, we construct a benchmark called TIGeR-Bench, including creative and knowledge-intensive domains, to standardize the evaluation of unified text-to-image generation and retrieval. Extensive experimental results on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our proposed method.