LoraHub：通过动态LoRA组合实现高效的跨任务泛化

摘要

低秩适应（LoRA）经常被用于微调大型语言模型（LLMs）以适用于新任务。本文研究了LoRA的组合性以实现跨任务泛化，并引入了LoraHub，这是一个战略框架，旨在有目的地组装在不同给定任务上训练的LoRA模块，以实现对未知任务的可适应性表现。通过从新任务中仅使用少量示例，LoraHub能够流畅地结合多个LoRA模块，消除了对人类专业知识的需求。值得注意的是，这种组合既不需要额外的模型参数，也不需要梯度。我们从Big-Bench Hard（BBH）基准测试中得出的实证结果表明，LoraHub可以有效地模拟在少样本场景中的上下文学习性能，无需在每个推断输入旁边提供上下文示例。我们研究的一个重要贡献是促进LoRA社区的发展，用户可以共享他们训练的LoRA模块，从而促进这些模块在新任务中的应用。我们预计这一资源将扩大对通用智能和生产中LLMs的访问，并推动进步。代码将在https://github.com/sail-sg/lorahub 上提供。

English

Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a strategic framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a novel task, LoraHub enables the fluid combination of multiple LoRA modules, eradicating the need for human expertise. Notably, the composition requires neither additional model parameters nor gradients. Our empirical results, derived from the Big-Bench Hard (BBH) benchmark, suggest that LoraHub can effectively mimic the performance of in-context learning in few-shot scenarios, excluding the necessity of in-context examples alongside each inference input. A significant contribution of our research is the fostering of a community for LoRA, where users can share their trained LoRA modules, thereby facilitating their application to new tasks. We anticipate this resource will widen access to and spur advancements in general intelligence as well as LLMs in production. Code will be available at https://github.com/sail-sg/lorahub.

LoraHub：通过动态LoRA组合实现高效的跨任务泛化

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

摘要

Support