語言模型可以控制自身的注意力

摘要

語言模型大部分的注意力只集中在上下文中很小的一部分，然而它們仍須讀取整個 KV 快取，才能找出少數重要的詞元。當使用者在 100 萬詞元的對話中詢問先前提及的細節時，全域注意力層在生成回覆的每個詞元時，都必須掃描完整的上下文。一種常見的緩解方法是透過輕量的代理分數預先選取相關詞元，但這種外在評分方式每一步仍須付出 O(N) 的成本。我們從一個簡單的問題出發，採取內在的觀點：模型難道不是本來就知道上下文中哪些部分是相關的嗎？為此，我們提出宣告式注意力（Declarative Attention, DA），這是一種引導模型在思維鏈中宣告其需要關注何處的機制，並將生成過程劃分為三種模式：<global>（全域）、<focus>（特定區域）與 <local>（僅限近期輸出）。推論引擎像解析工具呼叫一樣解析這些宣告，從而跳過大部分的 KV 快取讀取。在涵蓋 15 項長上下文任務的零樣本評估中，DA 應用於現成模型（Gemma-4-31B、Qwen-3.6-27B）在解碼期間分別減少了 52.0% 與 31.1% 的總關注詞元數，同時僅造成 1.27 個百分點與 2.75 個百分點的輕微準確度下降，且此下降幅度隨模型規模擴大而縮小。DA 為稀疏注意力開啟了一個新的軸向，未來在基於訓練的方法下仍有進一步的潛力有待探索。

English

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.