PEFT-Arena：從穩定性-可塑性視角理解參數高效微調

摘要

參數高效微調（PEFT）已成為調整大型語言模型的標準方法，然而現有評估主要側重於下游任務準確性，而忽略了對預訓練能力的保留。我們主張應透過穩定性-可塑性困境來評估PEFT：即目標任務適應性與抗遺忘能力之間的權衡。我們引入PEFT-Arena，這是一個同時衡量下游性能與通用能力保留的基準。在各種方法中，我們發現了不同的穩定性-可塑性特徵；在可比較的參數預算下，正交微調實現了最有利的帕累托前沿。為解釋這些差異，我們從兩個幾何角度分析PEFT更新。在權重空間中，頻譜分析揭示了參數化如何與預訓練奇異值結構相互作用。在激活空間中，保留指標顯示微調是保留還是扭曲了通用能力表徵，而遺忘與非等距表徵失真相關。最後，分析顯示最終的SFT檢查點往往超出更好的目標-保留操作點。受此啟發，我們展示了通過路徑式回退進行事後改進的案例研究。

English

Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.