ReRound：解决免校准大语言模型量化中点歧义的重建式舍入方法

摘要

ReRound（重建式舍入）是一种训练后量化方法，旨在解决标准最近舍入（RTN）方案在对接近量化区间中心的权重进行量化时所固有的中点模糊性问题。该方法从预训练的大语言模型（LLM）出发，训练一个条件扩散模型，为LLM的低比特权重生成连续重建。这些重建权重充当引导信号，用于消除接近区间中点的权重在舍入方向上的歧义。为了将这种重建引导的舍入与常规RTN相结合，ReRound引入了一种容差度量，用于衡量量化权重（而非最终量化整数）偏离中点的程度：位于中点附近容差区域内的量化权重，使用基于扩散的重建结果进行量化；而更接近量化边界的权重则使用RTN进行量化。通过扫描容差参数，ReRound生成多个候选的量化整数权重矩阵，并选择其反量化权重矩阵的前导奇异值与原始全精度权重最接近的那个候选。该选定候选决定了ReRound所使用的容差参数。 ReRound对较小规模的LLM尤为有效。在一系列此类模型上，它在3比特和4比特权重量化方面均稳定优于标准RTN。与大量无需校准的方法相比，ReRound取得了更优的精度，同时与依赖校准的方法保持竞争力，并且完全离线运行，在低比特推理过程中不引入任何额外开销。 ReRound策略为低比特量化提供了一种新方法。该方法也适用于LLM之外的AI模型。本文重点探讨其在小型LLM上的应用。

English

ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.