ReFreeKV：邁向無閾值之KV快取壓縮

摘要

為降低大型語言模型推理過程中的記憶體消耗，已有若干方法被提出用於KV快取剪枝。儘管這些技術能在許多資料集上實現無損記憶體壓縮，但它們通常依賴一個未被充分強調的條件：需預先設定輸入或領域特定的KV快取預算閾值，以達成最佳效能。然而，此類輸入敏感型設計在實際應用場景中可能受到顯著限制——由於開放領域的輸入涵蓋多樣化的領域、長度與難度層級，閾值的選取缺乏明確邊界。這使得此類輸入敏感閾值的依賴性成為根本限制，可能導致對任意輸入產生大幅效能衰退。本研究提出一項新型目標，解除穩健KV壓縮的閾值限制，倡導「無閾值」方法——在保留完整快取效能的同時自適應調整預算分配。我們進一步提出新穎方法ReFreeKV，作為此目標的首個實現方案。在橫跨13個資料集（涵蓋不同上下文長度、任務類型及模型規模）的實驗中，該方法驗證了其效能與效率。我們已於https://github.com/Patrick-Ni/ReFreeKV公開釋出程式碼。

English

To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection. As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for "threshold-free" methods that adaptively adjust budget allocation while preserving full-cache performance. We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective. Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency. Our code is publicly released at https://github.com/Patrick-Ni/ReFreeKV.