科學家的首次測驗：透過感知、理解與推理探測多模態大語言模型的認知能力

摘要

科學發現日益依賴於基於信息密集的科學數據和領域專業知識的複雜多模態推理。藉助專家級科學基準的賦能，科學多模態大語言模型（MLLMs）有望在實際工作流程中顯著提升這一發現過程。然而，當前的科學基準主要集中於評估MLLMs的知識理解能力，導致對其感知和推理能力的評估不足。為填補這一空白，我們提出了“科學家首次考試”（SFE）基準，旨在通過三個相互關聯的層次來評估MLLMs的科學認知能力：科學信號感知、科學屬性理解、科學比較推理。具體而言，SFE包含830個專家驗證的視覺問答對，涵蓋三種問題類型，跨越五個高價值學科的66個多模態任務。大量實驗表明，當前最先進的GPT-3和InternVL-3在SFE上的得分僅為34.08%和26.52%，凸顯了MLLMs在科學領域仍有顯著的提升空間。我們希望通過SFE獲得的洞見能夠推動AI增強科學發現的進一步發展。

English

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

科學家的首次測驗：透過感知、理解與推理探測多模態大語言模型的認知能力

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

摘要

Support