PEFT-Arena: 从稳定性-可塑性视角理解参数高效微调

摘要

参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）已成为适配大语言模型的标准方法，然而现有评估主要关注下游任务准确性，忽视了预训练能力的保持。我们认为，PEFT应当通过稳定性-可塑性困境（stability-plasticity dilemma）来评估，即目标任务适配与抗遗忘能力之间的权衡。为此，我们提出了PEFT-Arena基准，该基准同时衡量下游性能与通用能力保持。在不同方法中，我们发现其稳定性-可塑性特征存在显著差异；在可比参数预算下，正交微调（orthogonal fine-tuning）实现了最优的帕累托前沿（Pareto frontier）。为解释这些差异，我们从两个几何视角分析了PEFT的更新机制：在权重空间中，谱分析揭示了参数化方式如何与预训练奇异值结构相互作用；在激活空间中，保持性指标显示微调是否保留或扭曲了通用能力的表征，而遗忘与非线性表征扭曲相关。最后，分析表明最终的SFT检查点往往越过了一个更优的目标-保持性操作点。受此启发，我们展示了基于路径回退（path-wise rewinding）的后期改进案例研究。

English

Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.