MedAgent-Pro：基於多模態證據的醫療診斷推理代理工作流研究

MedAgent-Pro: Towards Multi-modal Evidence-based Medical Diagnosis via Reasoning Agentic Workflow

March 21, 2025

作者: Ziyue Wang, Junde Wu, Chang Han Low, Yueming Jin

cs.AI

摘要

開發可靠的AI系統以協助人類臨床醫生進行多模態醫療診斷，一直是研究人員的重要目標。近年來，多模態大型語言模型（MLLMs）在多個領域中獲得了顯著關注並取得了成功。憑藉其強大的推理能力以及根據用戶指令執行多樣化任務的能力，這些模型在提升醫療診斷方面展現出巨大潛力。然而，直接將MLLMs應用於醫療領域仍面臨挑戰。它們對視覺輸入的細緻感知能力不足，限制了其進行定量圖像分析的能力，而這對醫療診斷至關重要。此外，MLLMs在推理過程中常出現幻覺和不一致，而臨床診斷必須嚴格遵循既定標準。為應對這些挑戰，我們提出了MedAgent-Pro，這是一個基於證據的推理代理系統，旨在實現可靠、可解釋且精確的醫療診斷。該系統通過分層工作流程實現：在任務層面，基於知識的推理根據檢索到的臨床標準為特定疾病生成可靠的診斷計劃；而在案例層面，多個工具代理處理多模態輸入，根據計劃分析不同指標，並基於定量和定性證據提供最終診斷。在2D和3D醫療診斷任務上的全面實驗證明了MedAgent-Pro的優越性和有效性，而案例研究進一步凸顯了其可靠性和可解釋性。代碼可在https://github.com/jinlab-imvr/MedAgent-Pro獲取。

English

Developing reliable AI systems to assist human clinicians in multi-modal medical diagnosis has long been a key objective for researchers. Recently, Multi-modal Large Language Models (MLLMs) have gained significant attention and achieved success across various domains. With strong reasoning capabilities and the ability to perform diverse tasks based on user instructions, they hold great potential for enhancing medical diagnosis. However, directly applying MLLMs to the medical domain still presents challenges. They lack detailed perception of visual inputs, limiting their ability to perform quantitative image analysis, which is crucial for medical diagnostics. Additionally, MLLMs often exhibit hallucinations and inconsistencies in reasoning, whereas clinical diagnoses must adhere strictly to established criteria. To address these challenges, we propose MedAgent-Pro, an evidence-based reasoning agentic system designed to achieve reliable, explainable, and precise medical diagnoses. This is accomplished through a hierarchical workflow: at the task level, knowledge-based reasoning generate reliable diagnostic plans for specific diseases following retrieved clinical criteria. While at the case level, multiple tool agents process multi-modal inputs, analyze different indicators according to the plan, and provide a final diagnosis based on both quantitative and qualitative evidence. Comprehensive experiments on both 2D and 3D medical diagnosis tasks demonstrate the superiority and effectiveness of MedAgent-Pro, while case studies further highlight its reliability and interpretability. The code is available at https://github.com/jinlab-imvr/MedAgent-Pro.