当模态冲突：单模态推理不确定性如何主导多模态大模型的偏好动态

摘要

多模态大语言模型（MLLMs）在处理不同模态提供矛盾信息时必须解决冲突，这一过程我们称为模态追随。现有研究仅通过粗糙的数据集级统计量衡量该行为，忽略了模型在单模态推理中置信度的影响。本文提出新框架，将模态追随分解为两个基本要素：相对推理不确定性（单模态预测间针对具体案例的置信度差距）和固有模态偏好（不确定性平衡时模型的稳定偏向）。为验证框架，我们构建了可调控数据集，系统性地改变视觉与文本输入的推理难度。通过以熵作为细粒度不确定性度量，我们发现普遍规律：模型追随某一模态的概率随其相对不确定性的增加而单调递减。当模型以相近概率追随双模态的相对难度水平——即平衡点时，该指标可实际反映模型的固有偏好。与传统宏观比率不同，这种度量提供了更原理化、更少混杂的模态偏向表征方式，使其与单模态能力及数据集伪影解耦。进一步通过逐层预测探测，我们揭示了振荡的内部机制：在平衡点附近的模糊区域，模型会在不同层级间摇摆于双模态之间，这解释了外部观察到的犹豫现象。这些发现共同确立了相对不确定性与固有偏好作为模态追随的两大支配原则，为理解MLLMs如何解决冲突信息提供了量化框架与机制性见解。

English

Multimodal large language models (MLLMs) must resolve conflicts when different modalities provide contradictory information, a process we term modality following. Prior work measured this behavior only with coarse dataset-level statistics, overlooking the influence of model's confidence in unimodal reasoning. In this paper, we introduce a new framework that decomposes modality following into two fundamental factors: relative reasoning uncertainty (the case-specific confidence gap between unimodal predictions) and inherent modality preference( a model's stable bias when uncertainties are balanced). To validate this framework, we construct a controllable dataset that systematically varies the reasoning difficulty of visual and textual inputs. Using entropy as a fine-grained uncertainty metric, we uncover a universal law: the probability of following a modality decreases monotonically as its relative uncertainty increases. At the relative difficulty level where the model tends to follow both modalities with comparable probability what we call the balance point, a practical indicator of the model's inherent preference. Unlike traditional macro-level ratios, this measure offers a more principled and less confounded way to characterize modality bias, disentangling it from unimodal capabilities and dataset artifacts. Further, by probing layer-wise predictions, we reveal the internal mechanism of oscillation: in ambiguous regions near the balance point, models vacillate between modalities across layers, explaining externally observed indecision. Together, these findings establish relative uncertainty and inherent preference as the two governing principles of modality following, offering both a quantitative framework and mechanistic insight into how MLLMs resolve conflicting information.

当模态冲突：单模态推理不确定性如何主导多模态大模型的偏好动态

When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs

摘要

Support