透过表征层级解析剪枝生效的机制

摘要

网络剪枝通过移除次要参数或架构，常被期望在保持性能的同时提升效率。然而这种预期在语言任务中并不总成立：剪枝后的模型在非生成任务上表现良好，但在生成场景中往往失效。为理解这种差异，我们从表征层级视角分析网络剪枝，将语言模型的内部计算分解为三个连续空间：嵌入空间（隐藏表征）、逻辑值空间（Softmax前输出）和概率空间（Softmax后分布）。研究发现，嵌入空间和逻辑值空间的表征对剪枝引发的扰动具有较强鲁棒性，但从逻辑值到概率的非线性变换会放大这些偏差，这些偏差随时间步累积最终导致生成性能显著下降。相比之下，类别化词符概率子空间的稳定性与嵌入空间的鲁棒性共同支撑了剪枝在检索、多项选择等非生成任务中的有效性。我们的研究揭示了剪枝在不同任务中的差异化影响，并为其实际应用提供了指导。代码已开源：https://github.com/CASE-Lab-UMD/Pruning-on-Representations

English

Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network pruning from a representation-hierarchy perspective, decomposing the internal computation of language models into three sequential spaces: embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the embedding and logit spaces are largely robust to pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-token probability subspace, together with the robustness of the embedding space, supports the effectiveness of pruning for non-generative tasks such as retrieval and multiple-choice selection. Our analysis disentangles the effects of pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/Pruning-on-Representations

透过表征层级解析剪枝生效的机制

Demystifying When Pruning Works via Representation Hierarchies

摘要

Support