黎曼LoRA：一個無歧義LoRA優化的統一黎曼框架

摘要

低秩適應（LoRA）已成為大型語言模型（LLMs）參數高效微調的廣泛採用標準，顯著降低了記憶體和計算需求。然而，挑戰依然存在，包括尋找最佳初始化策略或緩解低秩矩陣分解中的過度參數化問題。在本研究中，我們提出了一種新穎的方法，在一個統一框架內同時解決這兩個挑戰。我們的方法將一組固定秩的LoRA矩陣視為一個光滑流形。將適配器視為該流形上的元素可消除過度參數化，而沿流形確定損失下降最快的方向則提供了初始化。我們特別注意使用數值線性代數和黎曼優化的最佳實踐，以實現數值穩定且計算高效的方法實現。在LLM和擴散模型架構上的實驗結果表明，與標準LoRA及其最先進的改進版本相比，黎曼LoRA在收斂速度和最終性能上均持續提升。

English

Low-Rank Adaptation (LoRA) has become a widely adopted standard for parameter-efficient fine-tuning of large language models (LLMs), significantly reducing memory and computational demands. However, challenges remain, including finding optimal initialization strategies or mitigating overparametrization in low-rank matrix factorization. In this work, we propose a novel approach that addresses both of the challenges simultaneously within a unified framework. Our method treats a set of fixed-rank LoRA matrices as a smooth manifold. Considering adapters as elements on this manifold removes overparametrization, while determining the direction of the fastest loss decrease along the manifold provides initialization. Special care is taken to obtain numerically stable and computationally efficient implementation of our method, using best practices from numerical linear algebra and Riemannian optimization. Experimental results on LLM and diffusion model architectures demonstrate that RiemannLoRA consistently improves both convergence speed and final performance over standard LoRA and its state-of-the-art modifications.

黎曼LoRA：一個無歧義LoRA優化的統一黎曼框架

RiemannLoRA: A Unified Riemannian Framework for Ambiguity-Free LoRA Optimization

摘要

Support