走向构建联邦GPT：联邦指导调整

Towards Building the Federated GPT: Federated Instruction Tuning

May 9, 2023

作者: Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, Yiran Chen

cs.AI

摘要

尽管“指导调整”的生成式大型语言模型（LLMs）展示了出色的泛化到新任务的能力，但训练阶段严重依赖大量多样且高质量的指导数据（如ChatGPT和GPT-4）。不幸的是，获取高质量数据，尤其是人工编写的数据，可能会在成本和获取方面带来重大挑战。此外，与隐私相关的担忧可能进一步限制对这些数据的访问，使得获取数据的过程变得复杂且微妙。因此，这阻碍了调整模型的泛化能力，可能会限制其在某些情境中的有效性。为了解决这一问题，我们的研究引入了一种名为联邦指导调整（FedIT）的新方法，该方法利用联邦学习（FL）作为LLMs指导调整的学习框架。这标志着首次探索了基于FL的LLMs指导调整。这一点尤为重要，因为文本数据主要由最终用户生成。因此，必须设计和调整FL方法，以有效利用这些用户在本地设备上存储的多样指导，同时保护隐私并确保数据安全。在本文中，通过进行广泛使用的GPT-4自我评估，我们展示了通过利用客户端端的异构和多样指导集合，结合提出的FedIT框架，相较于仅有有限本地指导的集中式训练，我们提高了LLMs的性能。此外，在本文中，我们开发了一个名为Shepherd的Github存储库。该存储库提供了一个探索使用跨不同类别的异构指导进行LLMs联邦微调的基础框架。

English

While ``instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amounts of diverse and high-quality instruction data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data, especially when it comes to human-written data, can pose significant challenges both in terms of cost and accessibility. Moreover, concerns related to privacy can further limit access to such data, making the process of obtaining it a complex and nuanced undertaking. Consequently, this hinders the generality of the tuned models and may restrict their effectiveness in certain contexts. To tackle this issue, our study introduces a new approach called Federated Instruction Tuning (FedIT), which leverages federated learning (FL) as the learning framework for the instruction tuning of LLMs. This marks the first exploration of FL-based instruction tuning for LLMs. This is especially important since text data is predominantly generated by end users. Therefore, it is imperative to design and adapt FL approaches to effectively leverage these users' diverse instructions stored on local devices, while preserving privacy and ensuring data security. In the current paper, by conducting widely used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with the proposed framework FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions. Further, in this paper, we developed a Github repository named Shepherd. This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.