從大型語言模型中提煉生物醫學知識：對不良藥物事件進行的案例研究

Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events

July 12, 2023

作者: Yu Gu, Sheng Zhang, Naoto Usuyama, Yonas Woldesenbet, Cliff Wong, Praneeth Sanapathi, Mu Wei, Naveen Valluri, Erika Strandberg, Tristan Naumann, Hoifung Poon

cs.AI

摘要

大型語言模型（LLMs），如GPT-4，展示了在包括健康應用在內的各種任務中卓越的能力。在本文中，我們研究了LLMs如何用於擴展生物醫學知識整理。我們發現，雖然LLMs已經在結構化生物醫學文本方面具有相當的能力，通過將其蒸餾成一個特定任務的學生模型，並通過自監督學習，可以實現比開箱即用的LLMs更大的收益，同時還具有成本、效率和白箱模型訪問等額外優勢。 我們對不良藥物事件（ADE）提取進行了一個案例研究，這是一個改善護理的重要領域。在標準ADE提取評估中，一個經過GPT-3.5蒸餾的PubMedBERT模型在不使用任何標記數據的情況下達到了與監督式最先進模型相當的準確性。儘管體積小了1000多倍，這個蒸餾模型在F1方面比其教師GPT-3.5高出超過6個絕對點，在GPT-4方面高出超過5個絕對點。 對蒸餾模型選擇（例如PubMedBERT vs BioGPT）和ADE提取架構進行的消融研究為生物醫學知識提取的最佳實踐提供了一些啟示。對其他標準生物醫學知識提取任務（如基因-疾病關聯和受保護健康信息）的蒸餾也實現了類似的收益，進一步說明了這種方法的潛力。

English

Large language models (LLMs), such as GPT-4, have demonstrated remarkable capabilities across a wide range of tasks, including health applications. In this paper, we study how LLMs can be used to scale biomedical knowledge curation. We find that while LLMs already possess decent competency in structuring biomedical text, by distillation into a task-specific student model through self-supervised learning, substantial gains can be attained over out-of-box LLMs, with additional advantages such as cost, efficiency, and white-box model access. We conduct a case study on adverse drug event (ADE) extraction, which is an important area for improving care. On standard ADE extraction evaluation, a GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised state-of-the-art models without using any labeled data. Despite being over 1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by over 6 absolute points in F1 and GPT-4 by over 5 absolute points. Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT) and ADE extraction architecture shed light on best practice for biomedical knowledge extraction. Similar gains were attained by distillation for other standard biomedical knowledge extraction tasks such as gene-disease associations and protected health information, further illustrating the promise of this approach.