Veagle：多模态表示学习的进展

Veagle: Advancements in Multimodal Representation Learning

January 18, 2024

作者: Rajat Chawla, Arkajit Datta, Tushar Verma, Adarsh Jha, Anmol Gautam, Ayush Vatsal, Sukrit Chaterjee, Mukunda NS, Ishaan Bhola

cs.AI

摘要

最近，人工智能领域的研究人员对语言和视觉如何结合产生了极大兴趣，这促使了多模态模型的发展，旨在无缝整合文本和视觉信息。多模态模型是大型语言模型（LLMs）的延伸，展现出在处理各种任务方面的显著能力，从图像字幕生成和视觉问答（VQA）到视觉定位。虽然这些模型展示了显著进展，但在准确解释图像并回答问题方面仍存在挑战，这在现实场景中很常见。本文介绍了一种增强现有模型多模态能力的新方法。针对当前视觉语言模型（VLMs）和多模态大型语言模型（MLLMs）存在的局限性，我们提出的Veagle模型融合了一种独特的机制，灵感来自先前作品的成功和见解。Veagle利用动态机制将编码的视觉信息直接投影到语言模型中。这种动态方法可以更细致地理解视觉环境中的复杂细节。为验证Veagle的有效性，我们在基准数据集上进行了全面实验，重点关注视觉问答和图像理解等任务。我们的结果显示，在性能方面有5-6%的改进，Veagle在很大程度上胜过现有模型。这些结果突显了该模型超越传统基准的多功能性和适用性。

English

Lately, researchers in artificial intelligence have been really interested in how language and vision come together, giving rise to the development of multimodal models that aim to seamlessly integrate textual and visual information. Multimodal models, an extension of Large Language Models (LLMs), have exhibited remarkable capabilities in addressing a diverse array of tasks, ranging from image captioning and visual question answering (VQA) to visual grounding. While these models have showcased significant advancements, challenges persist in accurately interpreting images and answering the question, a common occurrence in real-world scenarios. This paper introduces a novel approach to enhance the multimodal capabilities of existing models. In response to the limitations observed in current Vision Language Models (VLMs) and Multimodal Large Language Models (MLLMs), our proposed model Veagle, incorporates a unique mechanism inspired by the successes and insights of previous works. Veagle leverages a dynamic mechanism to project encoded visual information directly into the language model. This dynamic approach allows for a more nuanced understanding of intricate details present in visual contexts. To validate the effectiveness of Veagle, we conduct comprehensive experiments on benchmark datasets, emphasizing tasks such as visual question answering and image understanding. Our results indicate a improvement of 5-6 \% in performance, with Veagle outperforming existing models by a notable margin. The outcomes underscore the model's versatility and applicability beyond traditional benchmarks.