MM-Vet：評估大型多模型模型的整合能力

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

August 4, 2023

作者: Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang

cs.AI

摘要

我們提出了MM-Vet，一個評估基準，用於檢驗大型多模型（LMMs）在複雜多模任務上的表現。最近的LMMs展示了各種有趣的能力，例如解決黑板上的數學問題、推理新聞圖像中的事件和名人，以及解釋視覺笑話。快速的模型進展對評估基準的發展提出了挑戰。問題包括：（1）如何系統地結構和評估複雜的多模任務；（2）如何設計適用於各種問題和答案類型的評估指標；以及（3）如何提供超越簡單性能排名的模型洞察。為此，我們提出了MM-Vet，其設計基於一個洞察，即解決複雜任務的有趣能力通常是由一個通才模型能夠整合不同的核心視覺-語言（VL）能力而實現的。MM-Vet定義了6個核心VL能力，並檢驗了從能力組合中衍生的16個感興趣的整合。對於評估指標，我們提出了一個基於LLM的評估器，用於開放式輸出。該評估器使得能夠跨不同問題類型和答案風格進行評估，從而產生統一的評分指標。我們在MM-Vet上評估了代表性的LMMs，提供了有關不同LMM系統範式和模型能力的洞察。代碼和數據可在https://github.com/yuweihao/MM-Vet找到。

English

We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models. Code and data are available at https://github.com/yuweihao/MM-Vet.