在联邦学习中的低秩适应性选择性聚合

Selective Aggregation for Low-Rank Adaptation in Federated Learning

October 2, 2024

作者: Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, Liangqiong Qu

cs.AI

摘要

我们通过对学习的 A 和 B 矩阵的不对称性分析，研究了联邦学习中的 LoRA。在这个过程中，我们发现 A 矩阵负责学习通用知识，而 B 矩阵则专注于捕捉特定客户的知识。基于这一发现，我们引入了联邦共享低秩适应（FedSA-LoRA），该方法利用两个低秩可训练的矩阵 A 和 B 来建模权重更新，但只有 A 矩阵与服务器共享以进行聚合。此外，我们深入探讨了在其他 LoRA 变体（如 rsLoRA 和 VeRA）中学习的 A 和 B 矩阵之间的关系，揭示了一致的模式。因此，我们将我们的 FedSA-LoRA 方法扩展到这些 LoRA 变体，得到了 FedSA-rsLoRA 和 FedSA-VeRA。通过这种方式，我们建立了一个将 LoRA 与联邦学习相结合的通用范式，为未来关于结合联邦学习的后续 LoRA 变体的工作提供指导。在自然语言理解和生成任务上的大量实验结果表明了所提方法的有效性。

English

We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A and B matrices. In doing so, we uncover that A matrices are responsible for learning general knowledge, while B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned A and B matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method.