比較線性探測器與馬氏餘弦相似度

摘要

線性探針在可解釋性研究中被廣泛使用，且常透過餘弦相似度進行比較。馬氏距離餘弦相似度（MCS）是一種自然的任務感知優化，它透過測試資料共變異數重新加權內積。Ying等人（2026）報告指出，探針的MCS與在分布外（OOD）資料上訓練的參考探針之間，近乎完美地線性預測了該探針的OOD AUROC（R² = 0.98）。在此，我們將這項實證發現擴展至不同模型、層級與概念領域，並以封閉形式證明此普遍現象：對於投影呈高斯分佈的平衡類別，OOD AUROC與參考探針的MCS之所以呈線性關係，是因為兩者皆為測試資料上探針信噪比（SNR）的S形函數。該理論也預測了此線性關係何時失效，我們並以實證驗證之。MCS為比較線性探針提供了一個具理論基礎且實證有效的替代方案，以取代歐幾里得餘弦相似度。

English

Linear probes are widely used in interpretability research and often compared by cosine similarity. The Mahalanobis cosine similarity (MCS) between two directions, which reweights the inner product by test data covariance, is a natural task-aware refinement. Ying et al. (2026) report that a probe's MCS to a reference probe trained on the out-of-distribution (OOD) data near-perfectly linearly predicts the probe's OOD AUROC (R^2 = 0.98). Here, we extend this empirical finding across models, layers, and concept domains, and prove this general phenomenon in closed form: For balanced classes whose projections are Gaussian, OOD AUROC and MCS to the reference probe are linear because both are sigmoid-shaped functions of the probe's signal-to-noise ratio (SNR) on the test data. The theory also predicts when this linearity fails, which we verify empirically. MCS offers a theoretically grounded and empirically effective alternative to Euclidean cosine similarity for comparing linear probes.