GraLoRA：面向参数高效微调的细粒度低秩适配

摘要

低秩適應（LoRA）是一種廣受歡迎的生成模型參數高效微調（PEFT）方法，因其簡潔性和有效性而備受推崇。儘管近期有所改進，LoRA仍存在一個根本性限制：當瓶頸擴大時容易過擬合。它在秩為32-64時表現最佳，但在更高秩時其準確性停滯或下降，仍無法達到全量微調（FFT）的性能。我們發現其根本原因在於LoRA的結構性瓶頸，這會將不相關的輸入通道引入梯度糾纏，並扭曲梯度傳播。為解決這一問題，我們提出了一種新結構——粒度低秩適應（GraLoRA），它將權重矩陣劃分為子塊，每個子塊都有自己的低秩適配器。在幾乎不增加計算或存儲成本的情況下，GraLoRA克服了LoRA的侷限性，有效提升了表示能力，並更接近於FFT的行為。在代碼生成和常識推理基準測試中的實驗表明，GraLoRA始終優於LoRA及其他基線方法，在HumanEval+上實現了高達+8.5%的Pass@1絕對增益。這些改進在不同模型規模和秩設置下均保持一致，使GraLoRA成為一種可擴展且穩健的PEFT解決方案。代碼、數據和腳本可在https://github.com/SqueezeBits/GraLoRA.git獲取。

English

Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient fine-tuning (PEFT) of generative models, valued for its simplicity and effectiveness. Despite recent enhancements, LoRA still suffers from a fundamental limitation: overfitting when the bottleneck is widened. It performs best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks, still falling short of full fine-tuning (FFT) performance. We identify the root cause as LoRA's structural bottleneck, which introduces gradient entanglement to the unrelated input channels and distorts gradient propagation. To address this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA) that partitions weight matrices into sub-blocks, each with its own low-rank adapter. With negligible computational or storage cost, GraLoRA overcomes LoRA's limitations, effectively increases the representational capacity, and more closely approximates FFT behavior. Experiments on code generation and commonsense reasoning benchmarks show that GraLoRA consistently outperforms LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on HumanEval+. These improvements hold across model sizes and rank settings, making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts are available at https://github.com/SqueezeBits/GraLoRA.git

GraLoRA：面向参数高效微调的细粒度低秩适配

GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

摘要

Summary

Support

Support