BioMedLM：一种在生物医学文本上训练的包含 27 亿参数的语言模型

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

March 27, 2024

作者: Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, Christopher D. Manning

cs.AI

摘要

GPT-4和Med-PaLM 2等模型在各种生物医学自然语言处理任务上展现出令人印象深刻的性能。然而，这些模型拥有数千亿的参数，运行计算成本高昂，需要用户通过互联网发送其输入数据，并且是在未知数据源上训练的。更小、更有针对性的模型能否竞争呢？为了解决这个问题，我们构建并发布了BioMedLM，这是一个基于PubMed摘要和全文训练的、拥有27亿参数的GPT风格自回归模型。在微调后，BioMedLM能够产生强大的多项选择生物医学问答结果，与规模更大的模型竞争，比如在MedMCQA（开发）上取得了57.3%的得分，在MMLU医学遗传学考试上取得了69.0%的得分。BioMedLM还可以被微调以对医学话题上患者问题产生有用的回答。这表明更小的模型有可能作为特定自然语言处理应用的透明、保护隐私、经济和环保友好的基础，比如在生物医学领域。该模型可在Hugging Face Hub上获取：https://huggingface.co/stanford-crfm/BioMedLM。

English

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.