解锁材料科学的视觉记录：来自科学文献的大规模多模态数据集

摘要

材料科学文献中蕴藏着数十年的实验知识，这些知识以图形形式编码，然而这一视觉记录仍被锁定，无法被人工智能大规模获取。其核心难点在于结构层面：大多数科学图形是复合式的，单个图注同时描述多个子面板，导致直接的图像-文本配对不可靠。我们提出MatMMExtract，这是一个端到端的开源流水线，通过将复合图形分解为独立的子面板，并利用受材料科学分类体系指导的大型语言模型生成结构化的、有依据的注释，从而解决了这一问题。将该流水线应用于14810篇开放获取文章，MatMMExtract生成了MatSciFig数据集，包含来自180571个图形的391606个面板级图像-文本对，每个面板均配有子图注、涵盖19个大类和100多个子类型的两级可视化类别标注，以及科学摘要。为实现精准的面板定位，我们引入了MaterialScope，这是一个包含2811张手动标注的材料科学图形的领域专用检测数据集，在此数据集上微调的YOLO12-m检测器达到了0.9227的mAP_50。在六个基准语言模型中，Gemini 3.1 Flash Lite在注释生成方面实现了最佳的性价比，其输出中82%被评为良好，幻觉率为4.8%。基于MatSciFig的双编码器检索基线在R@1指标上相比零样本CLIP提升了4.4倍，证明了该数据集在视觉-语言学习中的直接应用价值。所有资源均向社区开放发布。

English

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.