当激活预言器学会不读取：微调预言器中的概念特定盲区

摘要

激活预言器（AOs）是经过训练的语言模型，用于回答关于另一模型内部激活的自然语言问题。它们提供了一种灵活的接口，用于从模型状态中读取隐藏信息，尤其是在相关信息在内部有所表征但在可见行为中缺失或不完整的情况下。然而，AO本身也是习得系统：其答案受训练数据、目标函数以及习得的报告行为所塑造，而非对所表征信息的中立读出。我们在一个受控的禁忌词猜测场景中对此进行了研究，其中主体模型经过微调，在内部使用一个隐藏概念的同时避免直接披露。与"在这种主体上训练的AO会成为专门的读取器"这一预期相反，我们发现微调后的AO可能成为特定概念的反向读取器：它们选择性地无法恢复在其自身训练期间持续存在的概念。这一失败并不能简单地用该概念在主体或预言器表征中的缺失来解释：目标在预言器内部仍是可解码的，而对数几率透镜（LogitLens）和层消融分析表明，该失败产生于AO的读出通路。我们的结果表明，行为泄漏、表征层面的可解码性以及AO的可言语化能力三者可能彼此分离，这对习得式可解释性接口提出了可靠性方面的担忧。

English

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.