數據與多模態大型語言模型之間的協同作用：從共同發展的角度進行調查

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

July 11, 2024

作者: Zhen Qin, Daoyuan Chen, Wenhao Zhang, Liuyi Yao, Yilun Huang, Bolin Ding, Yaliang Li, Shuiguang Deng

cs.AI

摘要

近年來，大型語言模型（LLMs）的快速發展已受到關注。基於強大的LLMs，多模態語言模型（MLLMs）將模態從文本擴展到更廣泛的領域，由於應用場景的擴大，吸引了廣泛關注。由於LLMs和MLLMs依賴大量的模型參數和數據來實現新興能力，數據的重要性正受到日益廣泛的關注和認可。追蹤並分析最近針對MLLMs的數據導向作品，我們發現模型和數據的發展並非兩條獨立的路徑，而是相互關聯的。一方面，更廣泛和高質量的數據有助於提升MLLMs的性能，另一方面，MLLMs可以促進數據的發展。多模態數據和MLLMs的共同發展需要清晰地了解：1）在MLLMs的哪個發展階段可以採用特定的數據導向方法以增強哪些能力，以及2）通過利用哪些能力並扮演哪些角色，模型可以為多模態數據做出貢獻。為了促進MLLM社區的數據-模型共同發展，我們從數據-模型共同發展的角度系統地回顧了與MLLMs相關的現有作品。與此調查相關的定期維護的項目可在https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md 上訪問。

English

The rapid development of large language models (LLMs) has been witnessed in recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from text to a broader spectrum of domains, attracting widespread attention due to the broader range of application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition. Tracing and analyzing recent data-oriented works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs, on the other hand, MLLMs can facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stage of MLLMs can specific data-centric approaches be employed to enhance which capabilities, and 2) by utilizing which capabilities and acting as which roles can models contribute to multi-modal data. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective. A regularly maintained project associated with this survey is accessible at https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.