多模態推理的問題感知視覺轉換器

摘要

視覺語言（VL）模型已獲得顯著的研究關注，實現了多模態推理的顯著進展。這些架構通常包括視覺編碼器、大型語言模型（LLM）和一個將視覺特徵與LLM表示空間對齊的投影模塊。儘管取得成功，但存在一個關鍵限制：視覺編碼過程與用戶查詢（通常以圖像相關問題的形式）仍然分離。因此，產生的視覺特徵可能未能最佳地調整為圖像的特定查詢元素。為了解決這個問題，我們引入了QA-ViT，一種用於多模態推理的問題感知視覺Transformer方法，直接將問題感知嵌入視覺編碼器。這種整合產生了動態的視覺特徵，專注於與提出的問題相關的圖像方面。QA-ViT是與模型無關的，可以有效地整合到任何VL架構中。大量實驗證明了將我們的方法應用於各種多模態架構的有效性，從而在不同任務中實現了一致的改進，展示了其增強視覺和場景文本理解能力的潛力。

English

Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

多模態推理的問題感知視覺轉換器

Question Aware Vision Transformer for Multimodal Reasoning

摘要

Support