随机注意力：重新思考KV缓存清除以实现高效推理

摘要

大语言模型在需要长程推理的任务上表现出色，但长思维链使 KV 缓存成为严重的内存瓶颈。现有 KV 缓存压缩方法都遵循同一范式：根据对每个缓存词元在后续计算中重要性的某种估计进行打分，然后保留得分最高的词元。我们发现，这种选择信号几乎没有贡献。随机注意力（Random Attention）保留提示词，并在每个注意力头内进行均匀随机逐出，完全不计算任何分数；在四个模型和六个推理任务上，它的表现与之前最强的逐出方法相当，而在 vLLM 部署中的吞吐量比后者高出 32–43%。受控实验揭示了其原因：1）提示词是缓存中脆弱的部分，不同选择器之间的差距大多只反映它们的选择信号是否恰好保留了提示词；2）推理轨迹凭借两个层面的冗余来抵御逐出——文本层面（模型在推理过程中会复述它仍需要的内容）和跨注意力头层面（每个头都保留自己的轨迹副本）。因此，只要提示词得到保留，随机抽取就能保留足够多模型仍需要的副本，不需要任何分数来选择它们。我们的代码见 https://github.com/SalesforceAIResearch/Random-Attention。

English

Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks it matches the strongest prior evictor while serving 32-43% higher throughput than it in vLLM deployment. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.