核心空间中精确高效的低秩模型融合

摘要

本文探讨了在合并大型神经网络低秩适配时面临的挑战。随着参数高效适配技术（如低秩适配LoRA）的兴起，模型微调变得更加便捷。尽管使用LoRA进行模型微调极为高效，现有合并方法往往通过合并全尺寸权重矩阵牺牲了这一效率。我们提出了核心空间合并框架，该框架能够在共同对齐基础上合并LoRA适配模型，从而在保持低秩适配效率的同时，显著提升跨任务准确率。我们进一步提供了形式化证明，表明投影至核心空间可确保信息无损，并通过复杂度分析展示了效率提升。大量实证结果表明，核心空间显著改进了现有合并技术，在视觉与语言任务上均取得了最先进的成果，同时仅消耗少量计算资源。代码库已发布于https://github.com/apanariello4/core-space-merging。

English

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

核心空间中精确高效的低秩模型融合

Accurate and Efficient Low-Rank Model Merging in Core Space

摘要

Support