MM-CRITIC：大型多模态模型作为多模态批判者的整体评估

摘要

批判能力对于模型实现自我提升并成为可靠的人工智能助手至关重要。尽管在纯语言环境中已得到广泛研究，但大型多模态模型（LMM）在多模态批判方面的探索仍然不足，尽管它们在图像描述和视觉推理等任务中的能力日益增强。本研究提出MM-CRITIC——一个从基础批判、修正批判与比较批判三个维度综合评估LMM批判能力的基准框架。该框架涵盖8种主要任务类型超过500项任务，收集了不同参数规模的LMM生成的4471个样本响应。为提升评估信度，我们通过专家指导构建标准答案评分体系，引导GPT-4o对模型响应进行标注并生成参考性批判意见，以此作为可靠评判的基准。大量实验验证了MM-CRITIC的有效性，并对主流LMM的多维度批判能力进行了全面评估。深入分析揭示了若干关键发现，包括响应质量与批判能力的相关性，以及不同评估维度下批判难度的差异性。相关代码已开源：https://github.com/MichealZeng0420/MM-Critic。

English

The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.

MM-CRITIC：大型多模态模型作为多模态批判者的整体评估

MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique

摘要

Support