重新审视显著图：构建认知对齐的解释性分类与评估框架

Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations

November 17, 2025

作者: Yehonatan Elisha, Seffi Cohen, Oren Barkan, Noam Koenigstein

cs.AI

摘要

显著性图谱在深度学习的视觉解释中应用广泛，但关于其设计初衷与多样化用户查询的匹配度，学界始终缺乏基本共识。这种模糊性严重制约了解释方法的有效评估与实际应用。为弥补这一空白，我们提出参照系×粒度（RFxG）分类法——一个基于原则的概念框架，该框架沿两个基本维度对显著性解释进行系统化组织：参照系维度区分单点式解释（"为何有此预测？"）与对比式解释（"为何是该结果而非替代项？"）；粒度维度涵盖从细粒度类别级（如"为何是哈士奇？"）到粗粒度组群级（如"为何是犬科？"）的解释层级。通过RFxG视角，我们揭示了现有评估指标的关键局限：这些指标过度侧重单点式忠实度，却忽视了对比推理与语义粒度。为系统评估解释质量在RFxG双维度的表现，我们提出四项新颖的忠实度度量指标。我们的综合评估框架将该指标体系应用于十种前沿显著性方法、四种模型架构及三个数据集。通过推动以用户意图为导向的评估范式转型，本研究不仅为开发忠于模型内在行为的视觉解释奠定了理论基础，更提供了关键实践工具，使解释结果能真正契合人类认知与探究活动的复杂性。

English

Saliency maps are widely used for visual explanations in deep learning, but a fundamental lack of consensus persists regarding their intended purpose and alignment with diverse user queries. This ambiguity hinders the effective evaluation and practical utility of explanation methods. We address this gap by introducing the Reference-Frame times Granularity (RFxG) taxonomy, a principled conceptual framework that organizes saliency explanations along two essential axes:Reference-Frame: Distinguishing between pointwise ("Why this prediction?") and contrastive ("Why this and not an alternative?") explanations. Granularity: Ranging from fine-grained class-level (e.g., "Why Husky?") to coarse-grained group-level (e.g., "Why Dog?") interpretations. Using the RFxG lens, we demonstrate critical limitations in existing evaluation metrics, which overwhelmingly prioritize pointwise faithfulness while neglecting contrastive reasoning and semantic granularity. To systematically assess explanation quality across both RFxG dimensions, we propose four novel faithfulness metrics. Our comprehensive evaluation framework applies these metrics to ten state-of-the-art saliency methods, four model architectures, and three datasets. By advocating a shift toward user-intent-driven evaluation, our work provides both the conceptual foundation and the practical tools necessary to develop visual explanations that are not only faithful to the underlying model behavior but are also meaningfully aligned with the complexity of human understanding and inquiry.