显性还是隐性？多模态临床AI逐模态失败分析的可复用框架

摘要

多模态临床模型通常以所有模态都存在时的准确率来评判，但部署时模态会缺失；在心电图是常规检查的地方，超声心动图往往不可用。于是，除了准确率损失的大小之外，还有两个问题更为关键：缺失的是哪个模态，以及一旦该模态被丢弃，模型是“响亮地”失败还是“静默地”失败。这一区分是逐样本且模态级的，并且不同于事后特征归因（如SHAP）。模型经常被替换；而回答这些问题的评估是复用的。我们提出一个模型无关的模态失败框架：给定N个模态嵌入、任意掩码感知探针和标签，它返回逐样本失败分类、一个将误差归因到模态的逐模态互补性矩阵，以及一个显性/静默失败对比的模态丢弃概况，仅使用部署时可观测的信号，就能将可监测的失败与那些在远离决策边界处未触发任何标志便通过（即实际错误）的失败区分开来。我们将其发布为一个小型、经过单元测试的工具包，并用植入的真值进行验证。在不同随机种子下，它能够恢复所植入的模态主导性和互补子集，报告各模态的显性/静默失败率，并可扩展到三模态互补矩阵；由于植入的结构是构造时已知的，这验证的是逐样本归因的恢复能力，而非临床性能。随后，我们在配对的MIMIC-IV队列上，将该框架实例化于冻结的EchoJEPA和HuBERT-ECG嵌入上，用于预测左心室射血分数（LVEF）和EF≤40%的HFrEF判定阈值；在留出测试集（n=245）上，丢弃超声心动图后错误率几乎翻倍。超声心动图与心电图之间狭窄的交集限制了队列规模，而这本身就是一个关于心脏基础模型的部署发现。我们的所有工作可在 https://github.com/criticaldata/PRIMED-AI 获取。

English

Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped. The distinction is per-example and modality-level, and is separate from post-hoc feature attribution (e.g. SHAP). Models are replaced often; the evaluation that answers these questions is reused. We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy, a per-modality complementarity matrix that attributes error to modalities, and a loud-vs-silent dropout profile separating monitorable failures from those that pass unflagged far from the decision boundary, using only deployment-observable signals. We release it as a small, unit-tested harness and validate it against planted ground truth. Across seeds it recovers that planted modality dominance and complementary subset, reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix; because the planted structure is known by construction, this validates recovery of per-example attribution rather than clinical performance. We then instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF and the EF <= 40% HFrEF gate over a paired MIMIC-IV cohort, where on the held-out test split (n = 245) dropping echo nearly doubles error. The narrow echo-to-ECG overlap that bounds cohort size is itself a deployment finding for cardiac foundation models. All of our work can be found at https://github.com/criticaldata/PRIMED-AI.