自動櫻桃採摘機：從由語言驅動的高質量生成數據中學習

Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language

June 28, 2024

作者: Yicheng Chen, Xiangtai Li, Yining Li, Yanhong Zeng, Jianzong Wu, Xiangyu Zhao, Kai Chen

cs.AI

摘要

擴散式模型在生成具有不同佈局的高品質圖像方面展現了巨大潛力，這有助於下游感知任務。然而，僅由語言驅動的完全自動佈局生成以及用於衡量多個生成實例的適當指標尚未得到很好的探索。在這項工作中，我們提出了Auto Cherry-Picker（ACP），這是一個新穎的框架，用於生成高質量的多模態訓練示例，以擴充感知和多模態訓練。從一個簡單的自然語言概念列表開始，我們提示大型語言模型（LLMs）生成詳細描述並設計合理的佈局。接下來，我們使用現成的文本到圖像模型生成多個圖像。然後，使用全面設計的指標對生成的數據進行精煉以確保質量。特別地，我們提出了一個新的指標，名為綜合佈局和圖像分數（CLIS），用於公平評估生成的圖像。我們的合成高質量示例通過定制初始概念列表，在各種情況下提升了性能，特別是在應對長尾分佈和不平衡數據集所帶來的挑戰方面。下游任務的實驗結果表明，Auto Cherry-Picker可以顯著提高現有模型的性能。此外，我們已徹底研究了CLIS與下游任務性能提升之間的相關性，我們發現更好的CLIS分數導致更好的性能。這一發現顯示了評估指標在各種視覺感知和MLLM任務中的潛力。代碼將可用。

English

Diffusion-based models have shown great potential in generating high-quality images with various layouts, which can benefit downstream perception tasks. However, a fully automatic layout generation driven only by language and a suitable metric for measuring multiple generated instances has not been well explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework that generates high-quality multi-modal training examples to augment perception and multi-modal training. Starting with a simple list of natural language concepts, we prompt large language models (LLMs) to generate a detailed description and design reasonable layouts. Next, we use an off-the-shelf text-to-image model to generate multiple images. Then, the generated data are refined using a comprehensively designed metric to ensure quality. In particular, we present a new metric, Composite Layout and Image Score (CLIS), to evaluate the generated images fairly. Our synthetic high-quality examples boost performance in various scenarios by customizing the initial concept list, especially in addressing challenges associated with long-tailed distribution and imbalanced datasets. Experiment results on downstream tasks demonstrate that Auto Cherry-Picker can significantly improve the performance of existing models. In addition, we have thoroughly investigated the correlation between CLIS and performance gains in downstream tasks, and we find that a better CLIS score results in better performance. This finding shows the potential for evaluation metrics as the role for various visual perception and MLLM tasks. Code will be available.