RestoreKV：在激进查询无关KV缓存驱逐下恢复全量缓存行为

摘要

查询无关的KV缓存驱逐会一次性压缩上下文，并将所得缓存复用于任意未来查询，但在严格的预算下性能可能大幅下降。现有方法主要改进保留哪些原始KV对。我们引入RestoreKV，通过在同一总KV预算下引入学习式恢复来补充这种基于选择的方法。我们的关键洞见在于，尽管驱逐所丢失的信息依赖于具体上下文，但生成其紧凑补充的机制可以跨上下文共享。在上下文预填充之后，少量恢复token在单次LoRA适配的传递中关注完整KV缓存，生成一个紧凑的、由上下文条件化的恢复缓存。基础重要性评分器和驱逐规则保持不变，适配器在之后的所有查询和解码中被禁用。RestoreKV通过从冻结的全缓存模型进行参数高效的自蒸馏来训练，仅优化0.4%的参数，且无需任务特定调优。在四种骨干模型和四个长上下文基准上，RestoreKV显著减少了压缩引起的性能退化。在Qwen3-4B上，它在跨五种基础驱逐方法的60个配对且预算匹配的设置中改善了59个；在5%预算下，它将RULER-4K上的KVzip从38.2提升至73.2。应用于KVzip+时，RestoreKV在KVPress基准的16倍压缩下达到86.4的RULER准确率，同时在32K上下文评估中增加不到0.5%的一次性缓存构建开销。我们的项目页面可在 https://paper.pnu-cvsp.com/RestoreKV/ 获取。

English

Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complements this selection-based formulation with learned restoration under the same total KV budget. Our key insight is that, although the information lost through eviction is context-specific, the mechanism for generating its compact complement can be shared across contexts. After context prefill, a few restore tokens attend to the full KV cache in a single LoRA-adapted pass, generating a compact, context-conditioned restore cache. The base importance scorer and eviction rule remain unchanged, and the adapters are disabled for all subsequent queries and decoding. RestoreKV is trained through parameter-efficient self-distillation from the frozen full-cache model, optimizing only 0.4% of the parameters and requiring no task-specific tuning. Across four backbones and four long-context benchmarks, RestoreKV substantially reduces compression-induced degradation. On Qwen3-4B, it improves 59 of 60 paired, budget-matched settings across five base eviction methods; at a 5% budget, it raises KVzip from 38.2 to 73.2 on RULER-4K. Applied to KVzip+, RestoreKV reaches 86.4 RULER accuracy at 16times compression on the KVPress Benchmark, while adding less than 0.5% one-time cache-construction overhead in a 32K-context evaluation. Our project page is available at https://paper.pnu-cvsp.com/RestoreKV/