LoLDU：通过下三角-对角-上三角分解进行低秩调整，用于参数高效微调。

LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning

October 17, 2024

作者: Yiming Shi, Jiwei Wei, Yujia Wu, Ran Ran, Chengwei Sun, Shiyuan He, Yang Yang

cs.AI

摘要

模型规模的快速增长需要大量的计算资源进行微调。现有方法，如低秩适应（LoRA），旨在解决全面微调中处理大量更新参数的问题。然而，LoRA利用随机初始化和优化低秩矩阵来近似更新权重，可能导致次优收敛和与全面微调相比的准确度差距。为了解决这些问题，我们提出了LoLDU，一种参数高效的微调（PEFT）方法，与常规PEFT方法相比，可将可训练参数减少2600倍，同时保持可比较的性能。LoLDU利用下-对角-上分解（LDU）来初始化低秩矩阵，以实现更快的收敛和正交性。我们专注于优化对角矩阵以进行缩放变换。据我们所知，LoLDU在所有PEFT方法中具有最少的参数。我们在4个指令遵循数据集、6个自然语言理解（NLU）数据集、8个图像分类数据集以及包含多种模型类型（LLaMA2、RoBERTa、ViT和Stable Diffusion）的图像生成数据集上进行了大量实验，提供了全面详细的分析。我们的开源代码可在https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}上获取。

English

The rapid growth of model scale has necessitated substantial computational resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA) has sought to address the problem of handling the large updated parameters in full fine-tuning. However, LoRA utilize random initialization and optimization of low-rank matrices to approximate updated weights, which can result in suboptimal convergence and an accuracy gap compared to full fine-tuning. To address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning (PEFT) approach that significantly reduces trainable parameters by 2600 times compared to regular PEFT methods while maintaining comparable performance. LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank matrices for faster convergence and orthogonality. We focus on optimizing the diagonal matrix for scaling transformations. To the best of our knowledge, LoLDU has the fewest parameters among all PEFT approaches. We conducted extensive experiments across 4 instruction-following datasets, 6 natural language understanding (NLU) datasets, 8 image classification datasets, and image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and Stable Diffusion), providing a comprehensive and detailed analysis. Our open-source code can be accessed at https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.