KVarN：變異數正規化KV快取量化減輕推理任務中的誤差累積

摘要

測試時擴展是一種增強大型語言模型推理能力的有效方法，但在長序列解碼過程中，由於KV快取不斷增長，會導致記憶體瓶頸。KV快取量化有助於改善此問題，但現有方法主要在前置填充（prefill）設定下評估，而自迴歸解碼（autoregressive decoding）下的誤差行為有所不同。我們證明，在後者模式下，量化誤差會隨著時間步累積，其主要原因來自於錯誤的token尺度。我們提出KVarN，一種無需校準的KV快取量化器，它先應用哈達瑪旋轉（Hadamard rotation），再對K與V矩陣的兩個軸進行雙尺度變異數正規化（dual-scaling variance normalization）。我們發現，此組合能修正離群token尺度誤差，並顯著減少相較於現有基線的誤差累積。KVarN在生成式基準測試（包括MATH500、AIME24與HumanEval）上，以2位元精度創下KV快取量化的最新技術水準。KVarN方法的vLLM實作可於 https://github.com/huawei-csl/KVarN 取得。

English

Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows. KV-cache quantization can help improve this, but current methods are evaluated under prefill-like settings and errors behave differently under autoregressive decoding. We show that in the latter regime, quantization errors accumulate across timesteps, driven primarily by incorrect token scales. We introduce KVarN, a calibration-free KV-cache quantizer that applies a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the K and V matrices. We find that this combination fixes outlying token-scale errors and substantially reduces error accumulation over existing baselines. KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. A vLLM implementation of the KVarN method is available at https://github.com/huawei-csl/KVarN