Debias-SparseGPT：面向大型语言模型的偏差感知剪枝

摘要

模型压缩技术（如剪枝和量化）促进了大型语言模型（LLMs）的高效部署与加速。然而，近期研究表明，权重稀疏化方法（如SparseGPT）可能放大模型中已有的偏见，导致输出因提示中的人物线索不同而产生显著差异。在本文中，我们提出Debias-SparseGPT，一种结合表征去偏的训练后剪枝方法，该方法利用基于人口统计学对照输入定义的二阶项进行去偏。我们在多种生成式大型语言模型上对所提方法进行了实证验证。在各种模型和稀疏度设置（25%、50%以及结构化2:4稀疏）下，与SparseGPT相比，Debias-SparseGPT在保持模型困惑度和零样本准确率的同时，持续降低了剪枝引发的偏见。在最严格的2:4结构化稀疏模式（该模式对模型质量的损害最为严重）下，使用长上下文、内容丰富的样本扩充校准集，可进一步改善下游性能和公平性。总体而言，Debias-SparseGPT在保持稀疏模型计算效率的同时，推进了偏见-性能的权衡优化。

English

Model compression techniques such as pruning and quantization facilitate the efficient deployment and acceleration of Large Language Models (LLMs). However, recent studies show that weight sparsification methods, such as SparseGPT, can amplify existing biases in models, with outputs varying significantly depending on persona cues in the prompt. In this paper, we introduce Debias-SparseGPT, a post-training pruning method incorporating representational debiasing using a second-order term defined over demographically contrasting inputs. We perform empirical validation of our method over a wide range of generative LLMs. Across models and sparsity regimes (25%, 50%, and structured 2:4 sparsity), Debias-SparseGPT consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy. Under the most restrictive 2:4 structured sparsity pattern, which most aggressively degrades model quality, augmenting the calibration set with long-context, content-rich examples further improves both downstream performance and fairness. Overall, Debias-SparseGPT advances the bias-performance trade-off while preserving the computational efficiency of sparse models.