HealthAgentBench：一個針對具有挑戰性前沿AI代理的逼真代理醫療保健環境統一基準測試套件

摘要

隨著AI代理在處理複雜且長期的推理任務上越來越強大，嚴謹且全面的評估對於衡量其在真實醫療應用中的進展至關重要。我們推出HealthAgentBench，這是一個包含54項醫療代理任務的評測套件，涵蓋7個類別，每個類別都有其獨特的環境。該評測套件涵蓋了病人就醫過程中的多種工作流程以及廣泛的數據模態。每項任務都旨在複製一個端到端的臨床工作流程：在僅給出最小程度指引的情況下，代理必須探索原始醫療數據，在複雜環境中運作，並執行超越簡單提示的多步驟解決方案。最終報告任務成功率，作為HealthAgentBench整體表現的單一可解釋指標。在對前沿代理進行HealthAgentBench評估時，我們發現整體任務成功率仍然偏低，凸顯了該套件的難度。最強且最具成本效益的代理Codex GPT-5.5，僅達到約42%的成功率。除了整體表現，HealthAgentBench還揭示了不同任務類別中細微的優劣勢。前沿代理在自動開發基於EHR數據的研究建模流程方面展現出潛力，但醫學影像處理仍特別具有挑戰性，尤其是對Claude Code模型而言，而Codex GPT-5.5則展現出新興能力。結合大搜索空間與組合推理需求的任務，對當前所有代理來說仍難以應對。綜合這些結果，HealthAgentBench提供了一個具有挑戰性且貼近真實的評測基準，未來仍有很大的進步空間。我們在https://github.com/microsoft/HealthAgentBench 公開發布我們的評測套件。

English

As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.