EXPL-FR：透過視覺-語言對齊解釋人臉辨識模型

摘要

深度人臉辨識（FR）模型已達到接近飽和的準確度，但仍不透明：實務人員無法詢問相似度分數是依賴哪些語義屬性。EXPL-FR 在 FR 模型自身的嵌入空間內回答此問題。一個輕量級適配器將視覺-語言模型（VLM）的影像編碼器與凍結的 FR 空間對齊，僅使用人臉影像訓練，從未使用文字。由於 VLM 的編碼器共享同一空間，相同的適配器也適用於文字編碼器，將 22 個類別中的 978 個屬性提示詞（亦可擴展）轉換為 FR 空間錨點，無需額外成本。我們並不假設此轉移有效：透過人臉驗證協議來衡量，並僅改變適配器的消融實驗來隔離其貢獻。並非每個概念都能存活，因為 FR 模型透過丟棄其必須跨身份驗證的因素來獲得其不變性。一種無標籤的可偵測性度量比較每個概念在 FR 空間與 VLM 空間中的可分離性，其中最具可偵測性的 100 個概念構成模型的可讀語義特徵簽章，其分離身份的效果優於完整詞彙。我們涵蓋四個 FR 骨幹網路和兩個 VLM 編碼器；EXPL-FR 無需存取架構，並支援身份層級、逐影像與差異化解釋。我們在三個監督設定下對屬性層級審查進行基準測試：人工標籤（當前實務）、VLM 偽標籤，以及我們完全由提示詞驅動的審查，並與真實驗證行為比較。在無標籤的情況下，提示詞驅動的審查根據測得的各族群 RFW 錯誤率對四個 FR 模型進行排序，並根據其真實驗證成本對受控屬性變更進行排序。

English

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.