Uni-SMART：通用科學多模態分析與研究轉換器

Uni-SMART: Universal Science Multimodal Analysis and Research Transformer

March 15, 2024

作者: Hengxing Cai, Xiaochen Cai, Shuwen Yang, Jiankun Wang, Lin Yao, Zhifeng Gao, Junhan Chang, Sihang Li, Mingjun Xu, Changxin Wang, Hongshuai Wang, Yongge Li, Mujie Lin, Yaqi Li, Yuqi Yin, Linfeng Zhang, Guolin Ke

cs.AI

摘要

在科學研究及其應用中，科學文獻分析至關重要，因為它使研究人員能夠建立在他人工作的基礎上。然而，科學知識的快速增長導致學術文章數量大幅增加，使深入文獻分析變得越來越具挑戰性和耗時。大型語言模型（LLMs）的出現提供了應對這一挑戰的新途徑。LLMs以其擅長總結文本的能力而聞名，被視為改善科學文獻分析的潛在工具。然而，現有的LLMs存在其局限性。科學文獻通常包含各種多模態元素，如分子結構、表格和圖表，這些對以文本為中心的LLMs來說很難理解和分析。這個問題指出了迫切需要新解決方案的需求，這些解決方案能夠充分理解和分析科學文獻中的多模態內容。為了滿足這一需求，我們提出了Uni-SMART（通用科學多模態分析和研究Transformer），這是一個專為深入理解多模態科學文獻而設計的創新模型。通過在幾個領域進行嚴格的定量評估，Uni-SMART展示了優於領先的以文本為中心的LLMs的性能。此外，我們的探索延伸到實際應用，包括專利侵權檢測和對圖表的細緻分析。這些應用不僅突顯了Uni-SMART的適應性，還展示了它改變我們與科學文獻互動方式的潛力。

English

In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly articles, making in-depth literature analysis increasingly challenging and time-consuming. The emergence of Large Language Models (LLMs) has offered a new way to address this challenge. Known for their strong abilities in summarizing texts, LLMs are seen as a potential tool to improve the analysis of scientific literature. However, existing LLMs have their own limits. Scientific literature often includes a wide range of multimodal elements, such as molecular structure, tables, and charts, which are hard for text-focused LLMs to understand and analyze. This issue points to the urgent need for new solutions that can fully understand and analyze multimodal content in scientific literature. To answer this demand, we present Uni-SMART (Universal Science Multimodal Analysis and Research Transformer), an innovative model designed for in-depth understanding of multimodal scientific literature. Through rigorous quantitative evaluation across several domains, Uni-SMART demonstrates superior performance over leading text-focused LLMs. Furthermore, our exploration extends to practical applications, including patent infringement detection and nuanced analysis of charts. These applications not only highlight Uni-SMART's adaptability but also its potential to revolutionize how we interact with scientific literature.