关注注意力头：多模态大语言模型的拓扑表示对齐

摘要

表征对齐通过将多模态大语言模型内部表示正则化为外部视觉编码器的表示，已成为提升其性能的有效方法。然而，现有方法通常对齐语言骨干网络的固定层，忽略了Transformer模型的细粒度结构。本文提出逐头表征对齐方法，该方法在单个注意力头层级强制执行跨模态对齐。我们的方法基于柏拉图表征假说，专注于跨模态保留表征的拓扑结构（即局部邻域关系）。遵循互K近邻对齐度量，我们引入一个对比目标作为匹配局部结构的可微代理。HeRA在多模态训练期间将此目标应用于大语言模型中特定注意力头，这些头根据MKNN度量的对齐分数被选中。反直觉的是，我们发现对齐最不对齐的头能带来最大收益。在多个MLLM和18个基准上的广泛评估表明，HeRA在具有挑战性的视觉中心任务上持续提升性能，并通过自然抑制对语言先验的过度依赖，作为视觉幻觉的有效正则化器。我们的代码已公开发布。

English

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.