VisualWebInstruct：透過網路搜尋擴展多模態指令數據

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

March 13, 2025

作者: Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, Wenhu Chen

cs.AI

摘要

視覺語言模型在許多以感知為核心的任務上取得了顯著進展，然而，由於缺乏高質量且多樣化的訓練數據，其在推理導向任務上的進展似乎受到限制。本研究旨在解決推理導向多模態數據集稀缺的問題。我們提出了VisualWebInstruct——一種新穎的方法，利用搜索引擎創建一個涵蓋數學、物理、金融、化學等多個學科領域的多樣化、高質量數據集。從精心挑選的30,000張種子圖像出發，我們運用Google圖片搜索來識別包含相似圖片的網站。我們收集並處理了超過70萬個獨特URL來源的HTML內容。通過內容提取、過濾與合成的流程，我們構建了一個包含約90萬個問答對的數據集，其中40%為視覺問答對，其餘為文本問答對。在VisualWebInstruct上微調的模型展現了顯著的性能提升：(1) 基於Llava-OV-mid的訓練在各基準測試中實現了10-20%的絕對分數提升，(2) 基於MAmmoTH-VL的訓練則獲得了5%的絕對提升。我們的最佳模型MAmmoTH-VL2在10B參數級別內，於MMMU-Pro-std（40.7%）、MathVerse（42.6%）及DynaMath（55.7%）上展現了領先的性能。這些卓越的成果凸顯了我們數據集在增強視覺語言模型處理複雜多模態任務推理能力方面的有效性。

English

Vision-Language Models have made significant progress on many perception-focused tasks, however, their progress on reasoning-focused tasks seem to be limited due to the lack of high-quality and diverse training data. In this work, we aim to address the scarcity issue of reasoning-focused multimodal datasets. We propose VisualWebInstruct - a novel approach that leverages search engine to create a diverse, and high-quality dataset spanning multiple disciplines like math, physics, finance, chemistry, etc. Starting with meticulously selected 30,000 seed images, we employ Google Image search to identify websites containing similar images. We collect and process the HTMLs from over 700K unique URL sources. Through a pipeline of content extraction, filtering and synthesis, we build a dataset of approximately 900K question-answer pairs, with 40% being visual QA pairs and the rest as text QA pairs. Models fine-tuned on VisualWebInstruct demonstrate significant performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain. Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath (55.7%). These remarkable results highlight the effectiveness of our dataset in enhancing VLMs' reasoning capabilities for complex multimodal tasks.