ARCHead：面向大语言模型输出头的激活度量残差修正

摘要

仅权重量化可大幅降低大型语言模型（LLM）Transformer模块的存储占用，但实际后端通常仍以BF16或FP16保留最终的语言建模头（LM-head）。朴素地量化该投影会强烈扰动词表logit分布。我们提出ARCHead，一种打包式LM-head压缩器，它结合了量化低秩核心、分组INT4残差，以及在由激活值导出的度量下拟合的低秩校正。ARCHead不存储稠密的BF16头，并将持久化LM-head存储降至原来的1/3.7至1/3.9。在Qwen3-8B-Base上，其存储占用仅为BF16头的25.6%，同时达到1.007的相对困惑度；而同等存储的朴素INT4为1.14-1.16。替换AWQ或bitsandbytes遗留的BF16头仅增加0.006-0.007的交叉熵，且在我们测量中吞吐量变化小于2%。因此，ARCHead通过压缩块量化器不会量化的大型输出投影，对它们形成补充。代码可在 https://github.com/suayptalha/archead 获取。

English

Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.