SeKV：基於分層語義記憶的解析度自適應KV快取用於長上下文大型語言模型推論

摘要

大型語言模型日益頻繁地處理長上下文，其中KV緩存成為主要的記憶體瓶頸：其大小隨序列長度線性增長，且需在解碼過程中全程保留，若無壓縮，完全在GPU上快取將因成本過高而不可行。現有的KV緩存壓縮方法難以在效率與忠實保留上下文之間取得平衡。標記驅逐會丟失資訊，而語義分組則在預填充階段固定壓縮決策；兩者都無法在生成過程中，當壓縮片段變得相關時，恢復其標記層級的細節。為了解決這個問題，我們提出SeKV，一種解析度自適應的語義KV緩存，它將上下文組織成熵引導的語義片段，並將其儲存在GPU-CPU記憶體層級中，而不丟棄任何資訊。每個片段在GPU上保留一個輕量級摘要向量用於粗略路由，並在CPU上保留一個低秩SVD基用於按需的標記層級重建。一個經過訓練的放大機制，在解碼過程中選擇性地展開與查詢相關的片段，從而實現精確檢索，而無需在GPU上完整實體化KV緩存。SeKV能實現自適應的標記層級重建，同時保持基礎LLM完全凍結，並僅增加少於0.05%的可訓練參數。在四個基準測試中，SeKV相較於最強的語義壓縮基線平均提升5.9%，同時在128K上下文長度下，相較於完整KV緩存減少53.3%的GPU記憶體用量。程式碼開源於 https://github.com/AmirAbaskohi/SeKV。

English

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.