SEA：监督式嵌入对齐用于MLLM中的标记级视觉-文本整合

SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

August 21, 2024

作者: Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Di Zhang, Baoqun Yin, Wentao Zhang

cs.AI

摘要

最近，多模态大型语言模型（MLLMs）展示了出色的感知和推理能力，通常由视觉编码器、适配器和大型语言模型（LLM）组成。适配器作为视觉和语言组件之间的关键桥梁。然而，使用图像级监督训练适配器通常会导致显著的不对齐，削弱了LLMs的能力并限制了多模态LLMs的潜力。为了解决这个问题，我们引入了监督嵌入对齐（SEA），这是一种利用视觉-语言预训练模型（如CLIP）的标记级对齐方法，通过对比学习将视觉标记与LLM的嵌入空间对齐。这种方法确保了视觉和语言表示的更一致整合，增强了多模态LLMs的性能和可解释性，同时保留了它们固有的能力。大量实验证明，SEA有效地改善了MLLMs，特别是对于较小的模型，而无需增加额外的数据或推理计算。SEA还为开发更通用和适应性强的解决方案以增强多模态系统奠定了基础。

English

Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable perceptual and reasoning abilities, typically comprising a Vision Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as the critical bridge between the visual and language components. However, training adapters with image-level supervision often results in significant misalignment, undermining the LLMs' capabilities and limiting the potential of Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment (SEA), a token-level alignment method that leverages vision-language pre-trained models, such as CLIP, to align visual tokens with the LLM's embedding space through contrastive learning. This approach ensures a more coherent integration of visual and language representations, enhancing the performance and interpretability of multimodal LLMs while preserving their inherent capabilities. Extensive experiments show that SEA effectively improves MLLMs, particularly for smaller models, without adding extra data or inference computation. SEA also lays the groundwork for developing more general and adaptable solutions to enhance multimodal systems.