基于Gist令牌的简化稀疏注意力

摘要

稀疏注意力可以降低长上下文推理的成本，但大多数变体都会引入新的架构组件。我们提出简化稀疏注意力（SSA），一种无需架构变更的更简洁的稀疏注意力方法。具体而言，我们首先在穿插有概要标记（gist tokens）的序列上进行持续预训练。我们照常优化标准的下一个标记损失函数，但概要标记通过注意力掩码限制语言模型可以关注上下文的哪些部分；这教会模型将每个片段的重要信息压缩到概要标记中。在推理阶段，SSA通过当前查询与少量概要标记之间的注意力对片段进行评分，通过重新引入对应的原始标记来选择性地展开得分最高的前k个片段。由于查询仅与概要标记进行评分，我们避免了与对整个KV缓存进行朴素评分相关的内存带宽开销，且无需像其他稀疏注意力方法那样依赖辅助KV缓存方法。在LongBench上，SSA在相同压缩比下始终优于压缩方法和推理时稀疏注意力基线。更引人注目的是，在检索增强生成中，经过持续预训练后，SSA的性能甚至能超过完全注意力方法5.7个百分点。我们将此归因于SSA选择性展开的能力，它能够将注意力集中在与查询相关的片段上，并有效滤除噪声。SSA还可扩展为层次化概要之概要变体（H-SSA），该变体在保持或提高高压缩比（高达32倍）下精度的同时，实现了对数线性解码复杂度。代码已开源：https://github.com/yuzhenmao/simplified-sparse-attention/。

English

Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler approach to sparse attention that requires no architectural changes. Concretely, we first perform continued pretraining on sequences interleaved with gist tokens. We optimize the standard next-token loss as usual, but the gist tokens use an attention mask to restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk's important information into the gist tokens. At inference time, SSA scores chunks via attention between the current query and the small set of gist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against the gist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the full KV cache, without requiring the auxiliary KV cache approach used by sparse attention methods. On LongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, in retrieval-augmented generation, SSA can even outperform full attention after continued pretraining by over 5.7 points. We attribute this to the ability of SSA's selective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. SSA further extends to a hierarchical gist-of-gist variant (H-SSA) that achieves log-linear decoding complexity while maintaining or improving accuracy at high compression ratios up to 32x. The code is available at https://github.com/yuzhenmao/simplified-sparse-attention/.