看见还是知道？多模态大语言模型中的视觉上下文敏感性

摘要

多模态大语言模型（MLLMs）通过将视觉输入与预训练语言模型的丰富先验知识相结合，取得了强劲的性能。然而，它们在以视觉为中心的任务上经常失败，尤其是在视觉证据与预训练知识相冲突时。我们使用两种诊断范式分别探究这些失败：（1）通过图像重建探测视觉信息是否可用；（2）测量多模态上下文敏感性，即模型在多大程度上遵循视觉上下文而非语言先验。为支持第二种范式，我们引入了WhatIfVis基准，涵盖五个粗粒度维度（时空、颜色、数量、大小和重量），其问题既可从图像也可从先验知识得到答案。我们的分析得出三个发现：（i）粗粒度的视觉证据得以保留，因为这些属性可以从冻结MLLM的最终层图像token中重建。因此，在这些属性问题上的失败指向感知后的利用问题，而非感知过程中视觉编码的退化。（ii）即使被明确指示使用或忽略视觉证据，基础模型（未在WhatIfVis上进行监督微调）仍表现出不稳定的视觉上下文敏感性。监督微调（SFT）改善了这种可控性并跨领域泛化，激活修补进一步将视觉-先验权衡定位到所有六个模型中架构特定的深度。（iii）视觉-先验权衡可沿学习到的向量进行控制。即使没有任何意图指令，应用这一导向向量也能提高相对于基础模型的可控性。综合来看，这些结果重新定位了瓶颈，表明对于我们所研究的粗粒度属性，MLLM编码了视觉证据，但无法可靠地控制对其的依赖。

English

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.