一体化：通用LoRA用于参数高效微调

摘要

我们提出了广义LoRA（GLoRA），这是一种用于通用参数高效微调任务的先进方法。在增强低秩适应性（LoRA）的基础上，GLoRA采用了一个广义提示模块来优化预训练模型的权重并调整中间激活，从而在不同任务和数据集上提供更灵活和强大的能力。此外，GLoRA通过采用可扩展的、模块化的、逐层结构搜索来实现高效的参数适应，学习每一层的单独适配器。源自统一的数学公式，GLoRA展现出强大的迁移学习、少样本学习和领域泛化能力，通过在权重和激活上增加额外维度来适应新任务。全面的实验证明，GLoRA在自然、专业和结构化基准测试中胜过所有先前方法，在各种数据集上以更少的参数和计算实现了更高的准确性。此外，我们的结构重参数化设计确保GLoRA不会产生额外的推断成本，使其成为资源有限应用的实用解决方案。代码可在以下链接找到：https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA。

English

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adjusts to new tasks through additional dimensions on weights and activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured benchmarks, achieving superior accuracy with fewer parameters and computations on various datasets. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code is available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

一体化：通用LoRA用于参数高效微调

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

摘要

Support