解鎖材料科學的視覺記錄：來自科學文獻的大規模多模態數據集

摘要

材料科学文献中的图表蕴含了数十年的实验知识，然而这一视觉记录仍被封闭，无法被人工智能大规模获取。其核心难点在于结构：大多数科学图表是复合型的，单个标题同时描述多个子图，导致直接的图文配对不可靠。我们提出了MatMMExtract，这是一个端到端的开源流水线，通过将复合图表分解为独立的子图，并利用由精选材料科学分类法指导的大型语言模型，生成结构化且基于事实的标注，从而解决了这一问题。将MatMMExtract应用于14,810篇开放获取论文，生成了MatSciFig数据集：从180,571个图表中提取出391,606组面板级别的图文对，每个图文对都配有子标题、涵盖19个大类和100多个子类的两级可视化类别标注，以及一份科学摘要。为了实现精确的面板定位，我们引入了MaterialScope——一个包含2,811张手动标注的材料科学图表的领域特定检测数据集，在该数据集上，经过微调的YOLO12-m检测器实现了0.9227的mAP_50。在六个基准语言模型中，Gemini 3.1 Flash Lite在标注生成方面实现了成本与质量的最佳平衡，82%的输出被评为良好，幻觉率为4.8%。在MatSciFig上建立的双编码器检索基线，其R@1指标相比零样本CLIP提升了4.4倍，证明了该数据集在视觉-语言学习中的直接效用。所有资源均已向社区开放发布。

English

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.