EchoPrime：一种多视频视图引导的视觉-语言模型，用于全面的心脏超声解读。

EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation

October 13, 2024

作者: Milos Vukadinovic, Xiu Tang, Neal Yuan, Paul Cheng, Debiao Li, Susan Cheng, Bryan He, David Ouyang

cs.AI

摘要

超声心动图是最广泛使用的心脏成像模式，捕获超声视频数据以评估心脏结构和功能。超声心动图中的人工智能（AI）有潜力简化手动任务，并提高可重复性和精度。然而，大多数超声心动图AI模型是单视图、单任务系统，不能综合利用完整检查期间捕获的多个视图的互补信息，因此导致性能和应用范围有限。为解决这一问题，我们引入EchoPrime，这是一个基于多视图、视图信息的、基于视频的视觉-语言基础模型，经过1200多万视频-报告对的训练。EchoPrime使用对比学习来训练一个统一的嵌入模型，适用于包括罕见和常见疾病和诊断在内的全面超声心动图研究中的所有标准视图的表示。然后，EchoPrime利用视图分类和视图信息解剖关注模型来加权视频特定解释，准确映射超声心动图视图与解剖结构之间的关系。通过检索增强解释，EchoPrime整合所有超声心动图视频的信息，并进行全面的临床超声心动图解释。在两个独立医疗系统的数据集中，EchoPrime在心脏形态和功能的23个不同基准测试中取得了最先进的性能，超越了任务特定方法和先前基础模型的性能。经过严格的临床评估，EchoPrime可以协助医生自动进行全面超声心动图的初步评估。

English

Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. Artificial intelligence (AI) in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.