隨機注意力：重新思考KV快取淘汰以實現高效推理

摘要

大型語言模型在需要延伸推理的任務上表現優異，但長鏈的思考過程使 KV 快取成為嚴重的記憶體瓶頸。現有的 KV 快取壓縮方法共享同一種範式：依據某種對未來重要性的估計，為每個被快取的 token 評分，並保留得分最高的 token。我們證明此選擇訊號幾乎沒有貢獻。隨機注意力（Random Attention）保留提示（prompt），並在每個注意力頭（attention head）內均勻隨機地逐出 token，完全不計算任何分數；在四個模型與六個推理任務上，它與先前最強的逐出方法表現相當，同時在 vLLM 部署中提供比其高出 32–43% 的吞吐量。控制實驗解釋了這一點，顯示：1) 提示是快取中較脆弱的部分，且不同選擇器之間的大部分差距，僅在於它們的選擇訊號是否恰好保留了提示；2) 推理軌跡以兩個層級的冗餘保護自身免受逐出：在文字層面（模型在推理過程中會重述它仍需要的內容），以及跨注意力頭層面（每個注意力頭各自保留一份軌跡副本）；因此，一旦提示安全無虞，隨機抽取便能保留足夠多模型仍需要的內容副本，無需任何分數來挑選它們。我們的程式碼公開於 https://github.com/SalesforceAIResearch/Random-Attention。

English

Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks it matches the strongest prior evictor while serving 32-43% higher throughput than it in vLLM deployment. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.

隨機注意力：重新思考KV快取淘汰以實現高效推理

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

摘要

Support