面向长推理的信息感知KV缓存压缩

摘要

近年来，大语言模型（LLMs）的推理能力取得了显著进展，这导致其在预填充和解码阶段的键值（KV）缓存规模日益增大。现有KV缓存压缩方法主要依赖注意力权重来估计令牌重要性。虽然注意力机制能有效捕捉上下文相关性，但它忽略了与预测不确定性和令牌信息量相关的互补信息论信号。本文从前瞻性视角重新审视令牌重要性，并提出一种名为“Forward Influence”的度量指标，用于衡量压缩后的令牌对后续上下文的影响程度。我们的分析表明：由注意力分数选中的令牌主要影响邻近上下文，而高预测不确定性相关的令牌则对较远的未来上下文展现出显著更强的影响力。基于这一发现，我们提出InfoKV——一种融合信息论信号的熵感知KV缓存压缩框架。该框架将令牌级预测不确定性与逐层表示演化相结合，并在推理过程中将由此产生的熵分数与注意力分数进行整合。在基于Llama-3.1、Llama-3.2和DeepSeek-R1的长上下文推理基准测试中，实验结果表明：无论是长预填充还是长解码场景，InfoKV均持续优于现有基于注意力的KV压缩方法。

English

Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce Forward Influence, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.