ShadowPEFT：面向参数高效微调的影子网络

摘要

参数高效微调（PEFT）通过仅训练少量任务特定参数并冻结预训练主干网络，显著降低了大型语言模型（LLM）全参数微调的训练成本。然而现有方法（如低秩自适应LoRA）通过向独立权重直接插入独立的低秩扰动来实现自适应，导致适配过程呈现局部参数化特性。我们提出ShadowPEPT——一种集中式PEFT框架，通过深度共享的阴影模块实现层级精调。该框架在每层Transformer中维护并行阴影状态，并通过循环演化逐步生成更丰富的隐状态。这一设计将适配机制从分布式权重空间扰动转变为共享的层空间精调过程。由于阴影模块与主干网络解耦，可实现跨层复用、独立预训练，并可选择分离部署模式，特别适用于边缘计算场景。在生成与理解基准测试中，ShadowPEFT在可训练参数量相当的情况下达到或超越了LoRA和DoRA的性能。针对阴影预训练、跨数据集迁移、参数扩展、推理延迟及系统级评估的进一步分析表明，集中式层空间适配是传统低秩PEFT的一种具有竞争力且灵活可扩展的替代方案。

English

Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

ShadowPEFT：面向参数高效微调的影子网络

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

摘要

Support