KronQ：基於克羅內克因子黑塞矩陣的大語言模型量化

摘要

後訓練量化（Post-training quantization, PTQ）是一種廣泛應用於壓縮大型語言模型（LLMs）而無需重新訓練的技術。現有的二階PTQ方法（包括GPTQ）僅基於輸入激活統計量構建量化目標，實質上假設所有輸出通道對逐層重建目標的貢獻均等。我們提出KronQ，這是一個透過將梯度共變異數引入量化流程來挑戰此假設的PTQ框架。在克氏因子（Kronecker-factored）海森近似下，量化損失同時取決於激活與梯度共變異數，而KronQ在兩個互補層級利用此特性：（1）KronQ引入雙向非相干處理，利用梯度共變異數將現有的輸入側隨機旋轉擴展至輸出維度，從而降低跨輸入與輸出維度的權重幅度變異；（2）KronQ推導出一種新的敏感度度量，用於層間混合精度分配，該度量由梯度與激活海森跡所驅動。值得注意的是，在LLaMA-3-70B模型的2位元權重量化情境中，當GPTQ與GPTAQ發散或產生退化量化（在WikiText-2上困惑度大於2000）時，KronQ達到了7.93的困惑度。

English

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.