MEDIC：朝向在臨床應用中評估LLM的全面框架

MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

September 11, 2024

作者: Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan

cs.AI

摘要

大型語言模型（LLMs）在醫療應用領域的快速發展促使人們呼籲進行全面評估，超越像USMLE這樣經常引用的基準，以更好地反映現實世界的表現。雖然現實世界的評估是評估效用價值的有價值指標，但往往落後於LLM演進的速度，可能導致部署後的研究結果過時。這種時間上的脫節需要進行全面的前期評估，以指導特定臨床應用的模型選擇。我們介紹了MEDIC，一個評估LLMs在臨床能力的五個關鍵維度上的框架：醫學推理、倫理和偏見、數據和語言理解、情境學習以及臨床安全。MEDIC具有一個新穎的交叉檢驗框架，量化LLM在覆蓋範圍和幻覺檢測等方面的表現，而無需參考輸出。我們應用MEDIC來評估LLMs在醫學問答、安全性、摘要、筆記生成以及其他任務上的表現。我們的結果顯示不同模型大小、基準模型與醫學微調模型之間的性能差異，並對需要特定模型優勢的應用的模型選擇產生影響，例如低幻覺或較低推論成本。MEDIC的多面評估揭示了這些性能折衷，彌合了理論能力與在醫療設置中的實際實施之間的差距，確保最有前途的模型被確定並適應於各種醫療應用。

English

The rapid development of Large Language Models (LLMs) for healthcare applications has spurred calls for holistic evaluation beyond frequently-cited benchmarks like USMLE, to better reflect real-world performance. While real-world assessments are valuable indicators of utility, they often lag behind the pace of LLM evolution, likely rendering findings obsolete upon deployment. This temporal disconnect necessitates a comprehensive upfront evaluation that can guide model selection for specific clinical applications. We introduce MEDIC, a framework assessing LLMs across five critical dimensions of clinical competence: medical reasoning, ethics and bias, data and language understanding, in-context learning, and clinical safety. MEDIC features a novel cross-examination framework quantifying LLM performance across areas like coverage and hallucination detection, without requiring reference outputs. We apply MEDIC to evaluate LLMs on medical question-answering, safety, summarization, note generation, and other tasks. Our results show performance disparities across model sizes, baseline vs medically finetuned models, and have implications on model selection for applications requiring specific model strengths, such as low hallucination or lower cost of inference. MEDIC's multifaceted evaluation reveals these performance trade-offs, bridging the gap between theoretical capabilities and practical implementation in healthcare settings, ensuring that the most promising models are identified and adapted for diverse healthcare applications.