用於推理模型的值感知隨機KV快取淘汰

摘要

推理模型通过扩展思维链提升了准确性，但其长输出造成了内存与计算瓶颈。KV缓存淘汰方法通过从缓存中移除不重要的键值对来降低开销，然而这类方法的准确性往往低于基于选择的稀疏注意力替代方案（后者保留完整KV缓存）。我们识别出决定KV缓存淘汰准确性的关键因素：首先，少量值状态具有异常大的幅度，移除它们会导致灾难性失败——模型进入重复推理循环；其次，在淘汰过程中引入随机性通过增加缓存多样性提升准确性。基于这些发现，我们提出值感知随机KV缓存淘汰方法（VaSE），这是一种无需训练的解决方案，既能保护大幅值状态，又能促进多样化的淘汰决策。在六项推理任务中，采用4倍KV缓存压缩的Qwen3模型在同等稀疏度下，其平均准确率高于当前最先进的稀疏注意力方法，同时比最强的淘汰方法高出4%以上。总体而言，VaSE弥合了效率与准确性之间的差距，支持FlashAttention2，并为推理模型实现静态内存占用。

English

Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.