低秩適應中的子空間混合

Mixture-of-Subspaces in Low-Rank Adaptation

June 16, 2024

作者: Taiqiang Wu, Jiahao Wang, Zhe Zhao, Ngai Wong

cs.AI

摘要

本文介紹了一種受子空間啟發的低秩適應（LoRA）方法，該方法在計算效率、易於實現並且適用於大型語言、多模態和擴散模型。首先，我們將LoRA的權重等效地分解為兩個子空間，並發現簡單地混合它們可以增強性能。為了研究這種現象，我們通過一個細粒度的子空間鏡頭重新審視它，顯示這種修改等效於使用一個固定的混合器來融合子空間。為了更靈活，我們聯合學習混合器和原始LoRA權重，將該方法稱為子空間混合LoRA（MoSLoRA）。MoSLoRA在不同模態的任務中始終優於LoRA，包括常識推理、視覺指導調整和主題驅動的文本到圖像生成，展示了其有效性和韌性。代碼可在https://github.com/wutaiqiang/MoSLoRA{github}找到。

English

In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of LoRA into two subspaces, and find that simply mixing them can enhance performance. To study such a phenomenon, we revisit it through a fine-grained subspace lens, showing that such modification is equivalent to employing a fixed mixer to fuse the subspaces. To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation, demonstrating its effectiveness and robustness. Codes are available at https://github.com/wutaiqiang/MoSLoRA{github}.