Logit贡献评分识别非字面检索头

摘要

在长上下文应用中，大型语言模型通常基于相关上下文片段的含义而非直接复制粘贴来合成答案。识别哪些注意力头执行此类合成对于解释长上下文模型行为至关重要。然而现有检测器在结构设计上会遗漏这些注意力头：它们奖励被关注标记与生成标记相匹配的头，这种字面复制标准仅捕获了注意力头读取的位置，却未能揭示其通过输出-值（OV）电路写入的内容——而正是这一机制实现了非字面检索。我们提出对数几率贡献评分（LOCOS），这是一种写入感知型检测器，通过将每个注意力头的OV电路输出投影到答案标记的解嵌入方向进行评分，并在单次前向传播中对比"针"与"非针"源位置。在三个模型系列（Qwen3、Gemma-3、OLMo-3.1）上，针对NoLiMa非字面检索基准，对排名靠前的LOCOS注意力头进行均值消融，在更少的头数量下即导致ROUGE-L指标崩塌；在Qwen3-8B上，消融50个头使ROUGE-L从0.401降至0.000，而最强的基线仍保留0.292。所选注意力头具有检索特异性：在相同消融下，参数回忆和算术推理仍保持基线水平。在Qwen3-8B上，相同消融也使MuSiQue从0.55降至0.08，BABI-Long从0.62降至0.20，而随机头对照组与基线差异保持在0.05以内。

English

In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.