Debias-SparseGPT：針對大型語言模型的偏差感知剪枝

摘要

剪枝和量化等模型壓縮技術促進了大型語言模型（LLMs）的高效部署與加速。然而，近期研究顯示，諸如SparseGPT這類權重稀疏化方法可能放大模型中既有的偏見，使輸出結果因提示中的身分提示線索而產生顯著差異。在本文中，我們提出Debias-SparseGPT，這是一種訓練後剪枝方法，透過定義於人口統計對比輸入上的二階項來實現表徵去偏。我們在廣泛的生成式大型語言模型上對我們的方法進行了實證驗證。在各種模型與稀疏度設定（25%、50%及結構化2:4稀疏性）下，相較於SparseGPT，Debias-SparseGPT在維持模型困惑度與零樣本準確率的同時，持續降低了剪枝引發的偏見。在最嚴格的2:4結構化稀疏模式下（此模式對模型品質的退化影響最為劇烈），以長上下文、內容豐富的範例擴充校準集，可進一步提升下游任務效能與公平性。整體而言，Debias-SparseGPT在偏見與效能之間取得了更佳的權衡，同時保留了稀疏模型的計算效率。

English

Model compression techniques such as pruning and quantization facilitate the efficient deployment and acceleration of Large Language Models (LLMs). However, recent studies show that weight sparsification methods, such as SparseGPT, can amplify existing biases in models, with outputs varying significantly depending on persona cues in the prompt. In this paper, we introduce Debias-SparseGPT, a post-training pruning method incorporating representational debiasing using a second-order term defined over demographically contrasting inputs. We perform empirical validation of our method over a wide range of generative LLMs. Across models and sparsity regimes (25%, 50%, and structured 2:4 sparsity), Debias-SparseGPT consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy. Under the most restrictive 2:4 structured sparsity pattern, which most aggressively degrades model quality, augmenting the calibration set with long-context, content-rich examples further improves both downstream performance and fairness. Overall, Debias-SparseGPT advances the bias-performance trade-off while preserving the computational efficiency of sparse models.