MiniGPT4-Video：通过交错的视觉-文本标记推进视频理解的多模态LLM

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

April 4, 2024

作者: Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, Mohamed Elhoseiny

cs.AI

摘要

本文介绍了MiniGPT4-Video，这是一种专为视频理解而设计的多模态大型语言模型（LLM）。该模型能够处理时间视觉和文本数据，从而擅长理解视频的复杂性。在MiniGPT-v2取得成功的基础上，该模型在将视觉特征转换为LLM空间方面表现出色，取得了在各种图像-文本基准测试上令人印象深刻的成果，本文将模型的能力扩展到处理一系列帧，使其能够理解视频。MiniGPT4-Video不仅考虑视觉内容，还融入了文本对话，使模型能够有效地回答涉及视觉和文本组件的查询。所提出的模型优于现有的最先进方法，在MSVD、MSRVTT、TGIF和TVQA基准测试上分别取得了4.22%、1.13%、20.82%和13.1%的增益。我们的模型和代码已在此处公开提供：https://vision-cair.github.io/MiniGPT4-video/

English

This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/