真正的多模態情境學習需要關注視覺上下文

True Multimodal In-Context Learning Needs Attention to the Visual Context

July 21, 2025

作者: Shuo Chen, Jianzhe Liu, Zhen Han, Yan Xia, Daniel Cremers, Philip Torr, Volker Tresp, Jindong Gu

cs.AI

摘要

基於強大語言架構構建的多模態大型語言模型（MLLMs），已實現了多模態情境學習（MICL）——即通過包含圖像、問題和答案的少量多模態示範來適應新任務。儘管在標準視覺語言數據集上顯示出顯著改進，當前的MLLMs在利用示範中的視覺信息方面仍存在困難。具體而言，這些模型往往忽視視覺線索，過度依賴文本模式，導致僅是文本模仿而非真正的多模態適應。這種行為使得MICL仍停留在單模態層面，大大限制了其實際應用價值。更重要的是，這一限制常被那些無需理解視覺背景的任務性能提升所掩蓋。因此，如何有效增強MICL能力並可靠評估其性能仍待深入探索。針對這些問題，我們首先引入了動態注意力重分配（DARA），這是一種高效的微調策略，通過重新平衡視覺與文本標記間的注意力，鼓勵模型關注視覺背景。此外，我們提出了TrueMICL，這是一個專為MICL設計的數據集，包含支持集和測試集，明確要求整合多模態信息——特別是視覺內容——以正確完成任務。大量實驗證明了我們整體方案的有效性，展示了在多模態情境學習能力上的實質性提升。代碼和數據集可在https://chenxshuo.github.io/true-micl-colm 獲取。

English

Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demonstrations consisting of images, questions, and answers. Despite showing noticeable improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations. Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation. This behavior makes MICL still unimodal and largely restricts its practical utility. More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual context. As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored. To address these issues, we first introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens. In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal information-particularly visual content-for correct task completion. Extensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the true multimodal in-context learning capabilities. Code and datasets are available at https://chenxshuo.github.io/true-micl-colm .