语言模型能够控制自身的注意力

摘要

语言模型将大部分注意力集中在上下文的很小一部分上，却仍需读取整个KV缓存来找出少数几个关键token。如果用户在百万token的对话中询问此前的一个细节，全局注意力层在生成回复的每一个token时都必须扫描完整上下文。一种主流方法通过轻量级代理分数预先筛选相关token来缓解这一成本，但这种外在评分在每一步仍会产生O(N)的开销。我们采用一种内在方法，其出发点是一个简单的问题：模型自身难道不知道上下文中哪些部分是相关的吗？为此，我们提出声明式注意力（DA）——一种引导模型在思维链中声明其所需要关注的区域的协议，将生成过程划分为三种模式：<global>（完整上下文）、<focus>（特定区域）和<local>（仅最近的输出）。推理引擎像解析工具调用一样解析这些声明，从而跳过大部分KV缓存读取。在15项长上下文任务的零样本评估中，DA在现成模型（Gemma-4-31B、Qwen-3.6-27B）上显著减少了解码过程中被关注token的总量（分别减少52.0%和31.1%），准确率仅小幅下降（1.27个百分点、2.75个百分点），且下降幅度随模型规模增大而缩小。DA为稀疏注意力解锁了一个新的维度，在基于训练的方法下还有更大潜力，可供未来工作探索。

English

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.