公平性剪枝：透過差異性激活在GLU-MLP層中定位群體偏差

摘要

本工作提出「公平性剪枝」（Fairness Pruning），一種輕量級的結構性介入方法，旨在管理並於未來緩解大型語言模型（LLMs）中的人口統計偏誤。作為該方法的基礎實證驗證，本工作聚焦於因果偏誤定位。透過最小對比提示詞對與推論時激活值擷取，該方法可識別在GLU架構中處理人口統計屬性時產生差異反應的神經元，並在down_proj輸入端評估其訊號。實證評估係於最高達30億參數的模型（Llama-3.2系列與Salamandra-2B）上進行，結合標準化基準評測與質性文本生成實驗。結果顯示，將所識別之神經元歸零會改變模型對相關人口統計變項的回應方式。然而，此介入並非產生扁平化的緩解效果，而是造成雙向的偏誤去穩定化：由於BiasScore為無符號指標，候選集合中混雜了促成刻板印象與對抗刻板印象的神經元，而對整體偏誤的淨效應取決於何種符號居於主導地位。此介入極具精準性：在Llama-3.2-1B中至多歸零40個神經元（低於總MLP寬度的0.031%），即可在推理與一般知識能力上達到99.49%的平均保留率。這些發現實證確認了人口統計偏誤處理與模型能力可運作於可分離的電路上，為從盲目歸零邁向方向性行為調變建立了方法論基礎。

English

This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs). As a foundational empirical validation of this method, this work focuses on causal bias localization. Using minimally contrastive prompt pairs and inference-time activation capture, the method identifies neurons that react differentially when processing demographic attributes in GLU architectures, evaluating the signal at the down_proj input. Empirical evaluation was conducted on models of up to 3 billion parameters (Llama-3.2 family and Salamandra-2B), combining standardized benchmark evaluation with qualitative text generation experiments. Results demonstrate that zeroing the identified neurons alters how the model responds to associated demographic variables. However, rather than producing flat mitigation, the intervention causes bidirectional bias destabilization: because BiasScore is unsigned, candidate sets mix neurons that push toward and against the stereotype, and the net effect on aggregate bias depends on which sign dominates. The intervention is extremely surgical: zeroing at most 40 neurons in Llama-3.2-1B (less than 0.031% of total MLP width) achieves a mean retention of 99.49% in reasoning and general knowledge capabilities. These findings empirically confirm that demographic bias processing and model capabilities operate on dissociable circuits, establishing the methodological foundations for transitioning from blind zeroing toward directional behavior modulation.