BioMedLM：一個在生物醫學文本上訓練的擁有 2.7B 參數的語言模型

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

March 27, 2024

作者: Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, Christopher D. Manning

cs.AI

摘要

像GPT-4和Med-PaLM 2這樣的模型在各種生物醫學自然語言處理任務上展現了令人印象深刻的表現。然而，這些模型擁有數千億個參數，運算成本高昂，需要用戶通過互聯網發送輸入數據，並且是在未知數據來源上訓練的。更小、更具針對性的模型能夠競爭嗎？為了回答這個問題，我們建立並發布了BioMedLM，一個擁有27億參數的GPT風格自回歸模型，僅在PubMed摘要和完整文章上訓練。在進行微調後，BioMedLM能夠產生強大的多項選擇生物醫學問答結果，與更大的模型競爭，例如在MedMCQA（dev）上達到57.3%的得分，以及在MMLU醫學遺傳學考試上達到69.0%。BioMedLM也可以進行微調，以對醫學話題上患者問題提供有用答案。這表明較小的模型有可能作為特定自然語言處理應用的透明、保護隱私、經濟和環保的基礎，例如在生物醫學領域。該模型可在Hugging Face Hub上找到：https://huggingface.co/stanford-crfm/BioMedLM。

English

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.