Logit貢獻評分識別非字面檢索頭

摘要

在長上下文應用中，大型語言模型經常從相關上下文的語意中整合出答案，而非逐字複製貼上。辨識哪些注意力頭（attention heads）執行此整合，對於解讀長上下文模型行為至關重要。然而，現有檢測器在建構上會遺漏這些注意力頭：它們獎勵那些所關注的詞元與生成詞元匹配的注意力頭——這是一種僅捕捉「頭在哪裡讀取」、卻忽略其透過輸出值（OV）電路「寫入什麼」的字面複製標準，而OV電路正是承載非字面檢索的機制。我們提出「對數貢獻評分法」（Logit-Contribution Scoring, LOCOS），一種感知寫入的檢測器，透過將每個注意力頭的OV電路輸出投影到答案詞元的去嵌入方向（unembedding direction），並在一次前向傳遞中對比「針」（needle）與「非針」（off-needle）來源位置。在三個模型家族（Qwen3、Gemma-3、OLMo-3.1）中，對NoLiMa非字面檢索基準測試的頂尖LOCOS注意力頭進行平均消融，在較少注意力頭數量下便使ROUGE-L崩潰，效果優於先前的注意力基礎檢測法；在Qwen3-8B上，消融50個注意力頭使ROUGE-L從0.401降至0.000，而最強的基線方法仍保留0.292。所選取的注意力頭具有檢索特異性：在相同消融下，參數記憶與算術推理能力仍維持基線水準。在Qwen3-8B上，相同的消融也導致MuSiQue從0.55降至0.08，BABI-Long從0.62降至0.20，而隨機注意力頭控制組則維持在基線的0.05範圍內。

English

In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.