時間是否佔有一席之地？時間感知頭部：語言模型如何回憶特定時間信息

Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information

February 20, 2025

作者: Yein Park, Chanwoong Yoon, Jungwoo Park, Minbyul Jeong, Jaewoo Kang

cs.AI

摘要

儘管語言模型提取事實的能力已得到廣泛研究，但其如何處理隨時間變化的事實仍未被充分探討。我們通過電路分析發現了「時間頭」（Temporal Heads），這些特定的注意力頭主要負責處理時間性知識。我們證實這些頭存在於多個模型中，儘管其具體位置可能有所不同，且其響應會根據知識類型及其對應年份而有所差異。禁用這些頭會削弱模型回憶特定時間知識的能力，同時保持其一般能力，而不影響時間無關性和問答性能。此外，這些頭不僅在數值條件（如「2004年」）下被激活，也在文本別名（如「在……年」）下被激活，表明它們編碼了超越簡單數值表示的時間維度。進一步地，我們通過展示如何通過調整這些頭的值來編輯時間性知識，擴展了我們發現的潛在應用。

English

While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads primarily responsible for processing temporal knowledge through circuit analysis. We confirm that these heads are present across multiple models, though their specific locations may vary, and their responses differ depending on the type of knowledge and its corresponding years. Disabling these heads degrades the model's ability to recall time-specific knowledge while maintaining its general capabilities without compromising time-invariant and question-answering performances. Moreover, the heads are activated not only numeric conditions ("In 2004") but also textual aliases ("In the year ..."), indicating that they encode a temporal dimension beyond simple numerical representation. Furthermore, we expand the potential of our findings by demonstrating how temporal knowledge can be edited by adjusting the values of these heads.