ShadowPEFT：用於參數高效微調的影網絡

摘要

參數高效微調（PEFT）通過僅訓練少量任務特定參數並凍結預訓練主幹網絡，大幅降低了大型語言模型（LLM）全參數微調的訓練成本。然而現有方法（如低秩適應LoRA）通過向各權重矩陣直接插入獨立的低秩擾動來實現適應，導致適應過程呈現局部參數化特性。本文提出集中式PEFT框架ShadowPEFT，其通過深度共享的陰影模組進行層級精煉。在每個Transformer層中，ShadowPEPT維持並行陰影狀態，通過疊代演化逐步生成更豐富的隱藏表徵。該設計將適應機制從分散的權重空間擾動轉變為共享的層空間精煉過程。由於陰影模組與主幹網絡解耦，既可跨層重複使用、獨立預訓練，也能以分離模式部署，特別適用於邊緣計算場景。在生成與理解任務的實驗表明，ShadowPEFT在可訓練參數量相當的情況下達到或超越LoRA與DoRA的性能。針對陰影預訓練、跨數據集遷移、參數擴展、推理延遲及系統級評估的進一步分析證實：集中式層空間適應是傳統低秩PEFT具備競爭力與靈活性的替代方案。

English

Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

ShadowPEFT：用於參數高效微調的影網絡

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

摘要

Support