ReRound：以重建式舍入解決免校準LLM量化中的中點歧義

摘要

ReRound（重建式捨入）是一種訓練後量化方法，旨在解決標準最近值捨入（round-to-nearest, RTN）機制在量化權重接近量化區間中心時所固有的中點模糊性問題。該方法從預訓練的大型語言模型（LLM）出發，訓練一個條件擴散模型，以產生該 LLM 低位元權重的連續重建；這些重建權重作為引導訊號，用以消除位於區間中點附近之權重的捨入方向歧義。為了將這種重建引導的捨入與傳統 RTN 整合，ReRound 引入一個容忍度指標，衡量量化權重（而非最終量化整數）偏離中點的程度：位於中點周圍容忍區域內的量化權重，使用基於擴散模型的重建來量化；而更接近量化邊界的權重則以 RTN 量化。透過掃描容忍度參數，ReRound 生成多個候選的量化整數權重矩陣，並選取其反量化後權重矩陣的主奇異值與原始全精度權重最為接近的候選，由此決定 ReRound 所用的容忍度參數。ReRound 對較小型的 LLM 尤其有效。在一系列此類模型中，它在 3 位元與 4 位元權重量化上持續優於標準 RTN。與大量的免校正方法相比，ReRound 能達到更優的準確度；與依賴校正的方法相比，也具備競爭力，且完全離線運作，在低位元推論期間不引入任何額外開銷。ReRound 策略代表了低位元量化的一種新途徑。該方法可應用於 LLM 以外的 AI 模型；本文則聚焦於其在小型 LLM 上的應用。

English

ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.