留意注意力頭：多模態大語言模型的拓撲表示對齊

摘要

表徵對齊已成為一種有效的方法，透過將多模態大型語言模型（MLLMs）的內部表徵正則化至外部視覺編碼器的表徵，從而提升其性能。然而，現有方法通常對齊語言骨幹模型的固定層級，忽略了Transformer模型的細粒度結構。在本研究中，我們提出逐注意力頭的表徵對齊（HeRA），這是一種在個別注意力頭層級強制執行跨模態對齊的方法。我們的方法奠基於柏拉圖式表徵假說，聚焦於跨模態保留表徵的拓撲結構（即其局部鄰域關係）。遵循相互K最近鄰（MKNN）對齊指標，我們引入一個對比目標，作為匹配局部結構的可微分代理。HeRA在多模態訓練期間，將此目標應用於LLM中特定的注意力頭，這些注意力頭是根據MKNN指標的對齊得分所選取。與直覺相反，我們發現對齊最不齊的頭部能帶來最大增益。跨越多個MLLMs及18個基準的廣泛評估顯示，HeRA在具挑戰性的視覺中心任務上持續提升性能，並透過自然抑制對語言先驗的過度依賴，有效作為對抗視覺幻覺的正則化手段。我們的程式碼已公開釋出。

English

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.