透過代碼引導的合成多模態數據生成，擴展文本豐富圖像的理解能力

Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation

February 20, 2025

作者: Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark

cs.AI

摘要

針對包含豐富文本的圖像（如圖表和文件）進行推理，是視覺語言模型（VLMs）的一項關鍵應用。然而，由於多樣化的文本豐富視覺語言數據的稀缺，VLMs在這些領域往往表現不佳。為應對這一挑戰，我們提出了CoSyn框架，該框架利用僅限文本的大型語言模型（LLMs）的編碼能力，自動生成合成文本豐富的多模態數據。給定描述目標領域的輸入文本（例如“營養成分標籤”），CoSyn會提示LLM生成用於渲染合成圖像的代碼（如Python、HTML、LaTeX等）。通過將底層代碼作為合成圖像的文本表示，CoSyn能夠再次依賴僅限文本的LLM生成高質量的指令微調數據。利用CoSyn，我們構建了一個包含40萬張圖像和270萬行視覺語言指令微調數據的數據集。在七個基準測試上的全面實驗表明，使用我們的合成數據訓練的模型在競爭性開源模型（包括Llama 3.2）中達到了最先進的性能，並超越了GPT-4V和Gemini 1.5 Flash等專有模型。此外，CoSyn還能生成合成指向數據，使VLMs能夠在輸入圖像中定位信息，展示了其在開發能夠在現實環境中行動的多模態代理方面的潛力。

English

Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.