KronQ：基于克罗内克因子化海森矩阵的大语言模型量化

摘要

训练后量化（PTQ）是一种广泛采用的、无需重新训练即可压缩大型语言模型（LLM）的技术。现有的二阶PTQ方法（包括GPTQ）仅从输入激活统计信息构建量化目标，实际上假设所有输出通道对逐层重建目标的贡献均等。我们提出KronQ，一种通过将梯度协方差引入量化流水线来挑战这一假设的PTQ框架。在基于克罗内克因子分解的海森近似下，量化损失同时依赖于激活协方差和梯度协方差，而KronQ在两个互补层面上利用了这一特性。（1）KronQ引入双向非相干处理，通过梯度协方差将现有的输入侧随机旋转扩展到输出维度，从而在输入和输出两个维度上降低权重幅值的方差。（2）KronQ推导出一种新的灵敏度指标，用于基于梯度和激活海森迹的层间混合精度分配。值得注意的是，在LLaMA-3-70B的2比特权重量化场景中，当GPTQ和GPTAQ发散或产生退化量化（在WikiText-2上困惑度超过2000）时，KronQ实现了7.93的困惑度。

English

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.