走向語言模型中的3D 分子-文本解讀

Towards 3D Molecule-Text Interpretation in Language Models

January 25, 2024

作者: Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, Qi Tian

cs.AI

摘要

語言模型（LMs）已在各個領域產生了深遠影響。然而，它們在理解3D分子結構方面的固有限制顯著地限制了它們在生物分子領域的潛力。為了彌補這一差距，我們專注於3D分子-文本解釋，並提出3D-MoLM：3D分子語言建模。具體而言，3D-MoLM通過為LM配備3D分子編碼器，使LM能夠解釋和分析3D分子。這種整合是通過3D分子-文本投影器實現的，橋接了3D分子編碼器的表示空間和LM的輸入空間。此外，為了增強3D-MoLM對跨模態分子理解和指導遵循的能力，我們精心策劃了一個以3D分子為中心的指導調整數據集--3D-MoIT。通過3D分子-文本對齊和3D分子中心指導調整，3D-MoLM建立了3D分子編碼器和LM的整合。它在下游任務上顯著超越了現有基準，包括分子-文本檢索、分子字幕生成，以及更具挑戰性的開放文本分子問答任務，特別專注於3D相關特性。

English

Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties.