SeKV：面向长上下文大语言模型推理的层级语义记忆型分辨率自适应KV缓存

摘要

大型语言模型越来越多地运行在长上下文中，此时KV缓存成为主要的内存瓶颈：其大小随序列长度线性增长，且需要在解码过程中全程保留，若不进行压缩，在GPU上进行完整缓存代价高昂。现有的KV缓存压缩方法难以在效率与忠实保留上下文之间取得平衡。令牌驱逐会丢弃信息，而语义分组在预填充阶段就固定了压缩决策；两者都无法在生成过程中，当压缩跨度中的令牌级细节变得相关时，将其恢复。为此，我们提出SeKV，一种分辨率自适应的语义KV缓存，它将上下文组织为熵引导的语义跨度，并存储在GPU-CPU内存层次结构中，不丢弃任何信息。每个跨度在GPU上保留一个轻量级摘要向量用于粗粒度路由，并在CPU上保留一个低秩SVD基用于按需的令牌级重建。一个经过训练的聚焦机制在解码过程中选择性地扩展与查询相关的跨度，从而在不将完整KV缓存加载到GPU的情况下实现精确检索。SeKV实现了自适应的令牌级重建，同时保持基础LLM完全冻结，且仅增加不到0.05%的可训练参数。在四个基准测试中，SeKV相比最强的语义压缩基线平均提升5.9%，同时在128K上下文下相比完整KV缓存减少53.3%的GPU内存。代码可在https://github.com/AmirAbaskohi/SeKV获取。

English

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.