Uni-SMART：通用科学多模态分析与研究变换器

Uni-SMART: Universal Science Multimodal Analysis and Research Transformer

March 15, 2024

作者: Hengxing Cai, Xiaochen Cai, Shuwen Yang, Jiankun Wang, Lin Yao, Zhifeng Gao, Junhan Chang, Sihang Li, Mingjun Xu, Changxin Wang, Hongshuai Wang, Yongge Li, Mujie Lin, Yaqi Li, Yuqi Yin, Linfeng Zhang, Guolin Ke

cs.AI

摘要

在科学研究及其应用中，科学文献分析至关重要，因为它使研究人员能够借鉴他人的工作。然而，科学知识的快速增长导致学术文章数量大幅增加，使深入文献分析变得越来越具有挑战性和耗时。大型语言模型（LLMs）的出现为解决这一挑战提供了新途径。LLMs以其强大的文本摘要能力而闻名，被视为改进科学文献分析的潜在工具。然而，现有的LLMs也存在局限性。科学文献通常包含各种多模态元素，如分子结构、表格和图表，这些元素对以文本为重点的LLMs来说很难理解和分析。这一问题迫切需要新的解决方案，能够充分理解和分析科学文献中的多模态内容。为了满足这一需求，我们提出了Uni-SMART（Universal Science Multimodal Analysis and Research Transformer），这是一种专为深入理解多模态科学文献而设计的创新模型。通过在多个领域进行严格的定量评估，Uni-SMART展示了比领先的以文本为重点的LLMs更优越的性能。此外，我们的探索还延伸到实际应用，包括专利侵权检测和图表的细致分析。这些应用不仅突显了Uni-SMART的适应性，也展示了它革新我们与科学文献互动方式的潜力。

English

In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly articles, making in-depth literature analysis increasingly challenging and time-consuming. The emergence of Large Language Models (LLMs) has offered a new way to address this challenge. Known for their strong abilities in summarizing texts, LLMs are seen as a potential tool to improve the analysis of scientific literature. However, existing LLMs have their own limits. Scientific literature often includes a wide range of multimodal elements, such as molecular structure, tables, and charts, which are hard for text-focused LLMs to understand and analyze. This issue points to the urgent need for new solutions that can fully understand and analyze multimodal content in scientific literature. To answer this demand, we present Uni-SMART (Universal Science Multimodal Analysis and Research Transformer), an innovative model designed for in-depth understanding of multimodal scientific literature. Through rigorous quantitative evaluation across several domains, Uni-SMART demonstrates superior performance over leading text-focused LLMs. Furthermore, our exploration extends to practical applications, including patent infringement detection and nuanced analysis of charts. These applications not only highlight Uni-SMART's adaptability but also its potential to revolutionize how we interact with scientific literature.