EchoPrime：一個多視頻視圖輔助的視覺語言模型，用於全面的心臟超聲解讀

EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation

October 13, 2024

作者: Milos Vukadinovic, Xiu Tang, Neal Yuan, Paul Cheng, Debiao Li, Susan Cheng, Bryan He, David Ouyang

cs.AI

摘要

超聲心動圖是最廣泛使用的心臟影像模式，捕獲超聲視頻數據以評估心臟結構和功能。人工智慧（AI）在超聲心動圖中有潛力優化手動任務，提高可重複性和精確性。然而，大多數超聲心動圖AI模型是單視圖、單任務系統，未綜合利用完整檢查期間捕獲的多個視圖的補充信息，導致性能和應用範圍有限。為解決此問題，我們引入EchoPrime，一種基於多視圖、視圖資訊的、基於視頻的視覺語言基礎模型，訓練超過1200萬個視頻-報告對。EchoPrime使用對比學習為全面超聲心動圖研究中的所有標準視圖訓練統一嵌入模型，包括罕見和常見疾病和診斷。然後，EchoPrime利用視圖分類和視圖資訊解剖關注模型，加權視頻特定解釋，準確映射超聲心動圖視圖與解剖結構之間的關係。通過檢索增強解釋，EchoPrime整合來自全面研究中所有超聲心動圖視頻的信息，執行全面臨床超聲心動圖解釋。在兩個獨立醫療系統的數據集中，EchoPrime在23個不同心臟形態和功能基準上實現了最先進的性能，超越了任務特定方法和先前基礎模型的性能。經過嚴格的臨床評估後，EchoPrime可以協助醫生對全面超聲心動圖進行自動初步評估。

English

Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. Artificial intelligence (AI) in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.