数据和多模态大型语言模型之间的协同作用：一项来自共同发展视角的调查

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

July 11, 2024

作者: Zhen Qin, Daoyuan Chen, Wenhao Zhang, Liuyi Yao, Yilun Huang, Bolin Ding, Yaliang Li, Shuiguang Deng

cs.AI

摘要

近年来，大型语言模型（LLMs）的快速发展已经引起了广泛关注。基于强大的LLMs，多模态语言模型（MLLMs）将模态从文本扩展到更广泛的领域，因具有更广泛的应用场景而受到广泛关注。由于LLMs和MLLMs依赖大量的模型参数和数据来实现新兴能力，数据的重要性正在受到越来越广泛的关注和认可。追踪和分析最近针对MLLMs的数据导向作品，我们发现模型和数据的发展并非两条分开的道路，而是相互交织的。一方面，更广泛和更高质量的数据有助于提升MLLMs的性能，另一方面，MLLMs可以促进数据的发展。多模态数据和MLLMs的共同发展需要清晰了解：1）在MLLMs的哪个发展阶段可以采用特定的数据中心方法来增强哪些能力，以及2）通过利用哪些能力并扮演哪些角色，模型可以为多模态数据做出贡献。为促进MLLM社区的数据-模型共同发展，我们从数据-模型共同发展的角度系统回顾了与MLLMs相关的现有作品。与此调查相关的一个定期维护的项目可在https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md访问。

English

The rapid development of large language models (LLMs) has been witnessed in recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from text to a broader spectrum of domains, attracting widespread attention due to the broader range of application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition. Tracing and analyzing recent data-oriented works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs, on the other hand, MLLMs can facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stage of MLLMs can specific data-centric approaches be employed to enhance which capabilities, and 2) by utilizing which capabilities and acting as which roles can models contribute to multi-modal data. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective. A regularly maintained project associated with this survey is accessible at https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.