MM-Ego：构建以自我为中心的多模态LLM模型

MM-Ego: Towards Building Egocentric Multimodal LLMs

October 9, 2024

作者: Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, Yinfei Yang

cs.AI

摘要

本研究旨在全面探索构建用于自我中心视频理解的多模态基础模型。为实现这一目标，我们从三个方面着手。首先，由于自我中心视频理解的问答数据匮乏，我们开发了一个数据引擎，基于人工注释数据，高效生成了700万个高质量的自我中心视频问答样本，视频长度从30秒到一小时不等。这是目前规模最大的自我中心问答数据集。其次，我们贡献了一个具有挑战性的自我中心问答基准，包含629个视频和7,026个问题，用于评估模型在识别和记忆不同长度视频中的视觉细节方面的能力。我们引入了一种新的去偏差评估方法，以帮助减轻模型评估中存在的不可避免的语言偏差。第三，我们提出了一种专门的多模态架构，采用了一种新颖的“记忆指针提示”机制。该设计包括一个全局窥视步骤，以获得对整个视频的总体理解并识别关键视觉信息，然后是一个回退步骤，利用关键视觉信息生成响应。这使模型能够更有效地理解扩展视频内容。凭借数据、基准和模型，我们成功构建了MM-Ego，一种自我中心多模态LLM，在自我中心视频理解方面表现出强大的性能。

English

This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we develop a data engine that efficiently generates 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long, based on human-annotated data. This is currently the largest egocentric QA dataset. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel "Memory Pointer Prompting" mechanism. This design includes a global glimpse step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we successfully build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.