MM-Vet：评估用于综合功能的大型多模态模型

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

August 4, 2023

作者: Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang

cs.AI

摘要

我们提出了MM-Vet，这是一个评估基准，用于检验大型多模态模型（LMMs）在复杂多模态任务上的表现。最近的LMMs展示了各种有趣的能力，比如解决写在黑板上的数学问题、推理新闻图片中的事件和名人，以及解释视觉笑话等。快速的模型进展给评估基准的发展带来了挑战。问题包括：（1）如何系统地构建和评估复杂的多模态任务；（2）如何设计能够适用于各种问题和答案类型的评估指标；以及（3）如何为模型提供超越简单性能排名的洞察力。为此，我们提出了MM-Vet，它基于这样一个观点设计，即解决复杂任务的有趣能力通常是由通用模型能够整合不同核心视觉-语言（VL）能力而实现的。MM-Vet定义了6种核心VL能力，并检查了从这些能力组合中得出的16种感兴趣的集成。对于评估指标，我们提出了一种基于LLM的评估器，用于开放式输出。该评估器能够跨不同问题类型和答案风格进行评估，从而产生统一的评分指标。我们在MM-Vet上评估了代表性的LMMs，为不同LMM系统范式和模型的能力提供了洞察。代码和数据可在https://github.com/yuweihao/MM-Vet获得。

English

We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models. Code and data are available at https://github.com/yuweihao/MM-Vet.