多模态大语言模型（MLLMs）深受模态偏差的影响。

MLLMs are Deeply Affected by Modality Bias

May 24, 2025

作者: Xu Zheng, Chenfei Liao, Yuqian Fu, Kaiyu Lei, Yuanhuiyi Lyu, Lutao Jiang, Bin Ren, Jialei Chen, Jiawen Wang, Chengxin Li, Linfeng Zhang, Danda Pani Paudel, Xuanjing Huang, Yu-Gang Jiang, Nicu Sebe, Dacheng Tao, Luc Van Gool, Xuming Hu

cs.AI

摘要

多模态大语言模型（MLLMs）的最新进展在整合文本与图像等多样化模态方面展现出令人瞩目的成果。然而，MLLMs深受模态偏差影响，往往过度依赖语言而未能充分利用视觉输入等其他模态。本立场论文主张，MLLMs深受模态偏差的深刻影响。首先，我们诊断了当前模态偏差的现状，揭示了其在各类任务中的具体表现。其次，我们提出了一套针对MLLMs模态偏差的系统研究路线图。再次，我们识别了MLLMs中模态偏差的关键因素，并为未来研究提供了可操作的建议以减轻其影响。为验证这些发现，我们进行了实验，展示了各因素的影响：1. 数据特性：语言数据紧凑且抽象，而视觉数据冗余且复杂，导致学习动态中固有的不平衡。2. 骨干能力失衡：预训练语言模型在MLLMs中的主导地位，导致对语言的过度依赖及对视觉信息的忽视。3. 训练目标：当前目标往往未能促进跨模态的均衡对齐，致使学习偏向语言捷径。这些发现强调了采用均衡的训练策略和模型架构以更好地整合MLLMs中多种模态的必要性。我们呼吁跨学科合作，共同应对这些挑战，推动MLLM研究的创新。本研究为MLLMs中的模态偏差提供了新视角，并为开发更稳健、更通用的多模态系统提供了洞见，助力向通用人工智能迈进。

English

Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images. MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs. This position paper argues that MLLMs are deeply affected by modality bias. Firstly, we diagnose the current state of modality bias, highlighting its manifestations across various tasks. Secondly, we propose a systematic research road-map related to modality bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and offer actionable suggestions for future research to mitigate it. To substantiate these findings, we conduct experiments that demonstrate the influence of each factor: 1. Data Characteristics: Language data is compact and abstract, while visual data is redundant and complex, creating an inherent imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The dominance of pretrained language models in MLLMs leads to overreliance on language and neglect of visual information. 3. Training Objectives: Current objectives often fail to promote balanced cross-modal alignment, resulting in shortcut learning biased toward language. These findings highlight the need for balanced training strategies and model architectures to better integrate multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle these challenges and drive innovation in MLLM research. Our work provides a fresh perspective on modality bias in MLLMs and offers insights for developing more robust and generalizable multimodal systems-advancing progress toward Artificial General Intelligence.