響亮還是沉默？多模態臨床AI中逐模態失敗分析的可重用框架

摘要

多模態臨床模型通常是在所有模態都存在的情況下以準確率來評判，但在實際部署時某些模態會被移除；例如在心電圖為常規檢查的場景中，超聲心動圖往往並不可用。此時，除了準確率損失的大小之外，還有兩個問題至關重要：是哪個模態造成了失敗；以及一旦該模態被移除，模型是「大聲地」失敗還是「無聲地」失敗。這種區分是逐樣本、按模態層級進行的，且有別於事後特徵歸因（如 SHAP）。模型經常被替換，而回答這些問題的評估方法可以被重複使用。我們提出一個模型無關的模態失敗框架：給定 N 個模態嵌入、任意可感知遮罩的探針以及標籤，該框架會返回逐樣本的失敗分類法、一個按模態將誤差歸因的互補性矩陣，以及一個「大聲 vs. 無聲」的模態丟棄概況，僅使用部署時可觀測的信號，將可監測的失敗與那些在遠離決策邊界處未被標記就通過的失敗區分開來。我們以一個小型、經單元測試的測試工具（harness）形式發布該框架，並用人工植入的基準真相進行驗證。在不同的隨機種子下，它能恢復所植入的模態主導性與互補子集，報告各模態的「大聲 vs. 無聲」比率，並能擴展到三模態的互補性矩陣；由於植入的結構在構造時即已知，這驗證的是逐樣本歸因的恢復能力，而非臨床表現。接著，我們在一個配對的 MIMIC-IV 隊列上，針對 LVEF 以及 EF ≤ 40% 的 HFrEF 判定門檻，將該框架實例化於凍結的 EchoJEPA 和 HuBERT-ECG 嵌入之上；在留出的測試分割（n = 245）中，移除超聲心動圖會使誤差幾乎翻倍。限制隊列規模的「超聲心動圖-心電圖」重疊狹小，這本身就是一個關於心臟基礎模型的部署發現。我們的所有工作可在 https://github.com/criticaldata/PRIMED-AI 取得。

English

Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped. The distinction is per-example and modality-level, and is separate from post-hoc feature attribution (e.g. SHAP). Models are replaced often; the evaluation that answers these questions is reused. We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy, a per-modality complementarity matrix that attributes error to modalities, and a loud-vs-silent dropout profile separating monitorable failures from those that pass unflagged far from the decision boundary, using only deployment-observable signals. We release it as a small, unit-tested harness and validate it against planted ground truth. Across seeds it recovers that planted modality dominance and complementary subset, reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix; because the planted structure is known by construction, this validates recovery of per-example attribution rather than clinical performance. We then instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF and the EF <= 40% HFrEF gate over a paired MIMIC-IV cohort, where on the held-out test split (n = 245) dropping echo nearly doubles error. The narrow echo-to-ECG overlap that bounds cohort size is itself a deployment finding for cardiac foundation models. All of our work can be found at https://github.com/criticaldata/PRIMED-AI.