當模態衝突時：單模態推理不確定性如何主導多模態大語言模型的偏好動態

摘要

多模態大型語言模型（MLLMs）在處理不同模態提供矛盾資訊時必須解決衝突，此過程我們稱之為模態追隨。過往研究僅以粗糙的資料集層級統計數據衡量此行為，忽略了模型在單模態推理中置信度的影響。本文提出新框架，將模態追隨分解為兩個核心因子：相對推理不確定性（單模態預測間針對具體案例的置信度差距）與固有模態偏好（不確定性平衡時模型的穩定偏向）。為驗證此框架，我們構建了可控制資料集，系統性調節視覺與文字輸入的推理難度。透過以熵作為細粒度不確定性指標，發現普遍規律：模型遵循某模態的機率會隨其相對不確定性增加而單調遞減。當模型以相近機率遵循雙模態的相對難度水平——即平衡點時，可作為衡量模型固有偏好的實用指標。相較傳統宏觀比例，此測量方式能更系統性地表徵模態偏向，使其與單模態能力及資料集人為因素解耦。進一步透過層級預測探測，我們揭示振盪的內部機制：在平衡點附近的模糊區域，模型會在不同層間對模態搖擺不定，從而解釋外部觀測到的決策猶豫現象。這些發現共同確立了相對不確定性與固有偏好作為模態追隨的兩大支配原則，不僅提供量化框架，更從機制層面揭示MLLMs如何化解矛盾資訊。

English

Multimodal large language models (MLLMs) must resolve conflicts when different modalities provide contradictory information, a process we term modality following. Prior work measured this behavior only with coarse dataset-level statistics, overlooking the influence of model's confidence in unimodal reasoning. In this paper, we introduce a new framework that decomposes modality following into two fundamental factors: relative reasoning uncertainty (the case-specific confidence gap between unimodal predictions) and inherent modality preference( a model's stable bias when uncertainties are balanced). To validate this framework, we construct a controllable dataset that systematically varies the reasoning difficulty of visual and textual inputs. Using entropy as a fine-grained uncertainty metric, we uncover a universal law: the probability of following a modality decreases monotonically as its relative uncertainty increases. At the relative difficulty level where the model tends to follow both modalities with comparable probability what we call the balance point, a practical indicator of the model's inherent preference. Unlike traditional macro-level ratios, this measure offers a more principled and less confounded way to characterize modality bias, disentangling it from unimodal capabilities and dataset artifacts. Further, by probing layer-wise predictions, we reveal the internal mechanism of oscillation: in ambiguous regions near the balance point, models vacillate between modalities across layers, explaining externally observed indecision. Together, these findings establish relative uncertainty and inherent preference as the two governing principles of modality following, offering both a quantitative framework and mechanistic insight into how MLLMs resolve conflicting information.

當模態衝突時：單模態推理不確定性如何主導多模態大語言模型的偏好動態

When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs

摘要

Support