看見還是知道？多模態大型語言模型中的視覺情境敏感性

摘要

多模態大型語言模型（MLLMs）通過將視覺輸入與預訓練語言模型的豐富先驗知識相結合，展現出強勁的性能。然而，在以視覺為中心的任務上，它們經常表現不佳，特別是在視覺證據與預訓練知識發生衝突時。我們通過兩種診斷範式分別探討這類失敗：（1）透過圖像重建探查視覺資訊是否可用；（2）衡量多模態語境敏感度，即模型在多大程度上遵循視覺語境而非語言先驗。為支撐第二種範式，我們提出了 WhatIfVis 基準測試，涵蓋五個粗粒度維度（時空、顏色、數量、大小與重量），其問題可分別從圖像或先驗知識中獲得答案。我們的分析得出三項發現：（i）粗粒度的視覺證據得以保留，因為這些屬性可以從凍結 MLLMs 的最終層圖像 token 中重建出來。因此，在針對這些屬性的問題上出現的失敗，指向的是感知後的利用階段，而非感知過程中視覺編碼的退化。（ii）即使在被明確指示使用或忽略視覺證據時，未經微調的基礎模型（未在 WhatIfVis 上進行監督式微調）仍表現出不穩定的視覺語境敏感度。監督式微調（SFT）提升了這種可控性，並能跨領域泛化；激活修補進一步將視覺與先驗之間的權衡定位於所有六個模型中架構特定的深度層。（iii）視覺與先驗之間的權衡可沿著一個學習到的向量加以控制。應用此引導向量，即使在沒有任何意圖指令的情況下，也能提升相對於基礎模型的可控性。綜合來看，這些結果重新定位了瓶頸所在，表明對於我們所研究的粗粒度屬性，MLLMs 能夠編碼視覺證據，但無法可靠地控制對其依賴程度。

English

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.