將預訓練擴展至一千億條視覺語言模型的規模

Scaling Pre-training to One Hundred Billion Data for Vision Language Models

February 11, 2025

作者: Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, Xiaohua Zhai

cs.AI

摘要

我們對視覺-語言模型進行了前所未有規模的實證研究，使用了一千億個範例。我們發現，在許多常見的西方中心分類和檢索基準上，例如 COCO 圖片標註，模型性能在這個規模上趨於飽和。然而，具有文化多樣性的任務從這一千億規模的網絡數據中獲得了更實質的收益，這要歸功於其對長尾概念的覆蓋。此外，我們分析了模型的多語能力，展示了在資源稀缺語言中的收益。此外，我們觀察到，通過質量篩選（例如使用 CLIP）來減少預訓練數據集的大小，通常用於增強性能，可能會無意中減少即使在大規模數據集中也代表的文化多樣性。我們的結果凸顯了，儘管傳統基準在將嘈雜、原始網絡數據擴展到一千億個範例時可能不會從中受益顯著，但這種數據規模對於構建真正包容的多模態系統至關重要。

English

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented even in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.