在核心空間中實現精確且高效的低秩模型合併

摘要

本文探討了合併大型神經網絡低秩適應所面臨的挑戰。隨著參數高效適應技術（如低秩適應LoRA）的興起，模型微調變得更加便捷。雖然使用LoRA進行模型微調極為高效，但現有的合併方法往往通過合併完整大小的權重矩陣來犧牲這一效率。我們提出了核心空間合併框架，該框架能在共同的對齊基礎上合併LoRA適應的模型，從而保持低秩適應的效率，同時大幅提升跨任務的準確性。我們進一步提供了形式化證明，表明投影到核心空間可確保信息無損，並提供了複雜度分析，展示了效率提升。大量實驗結果表明，核心空間顯著改進了現有的合併技術，在視覺和語言任務上均達到了最先進的成果，同時僅使用了極少的計算資源。代碼庫可在https://github.com/apanariello4/core-space-merging獲取。

English

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

在核心空間中實現精確且高效的低秩模型合併

Accurate and Efficient Low-Rank Model Merging in Core Space

摘要

Support