通过基于检索的蒸馏来训练任务专家

Training Task Experts through Retrieval Based Distillation

July 7, 2024

作者: Jiaxin Ge, Xueying Jia, Vijay Viswanathan, Hongyin Luo, Graham Neubig

cs.AI

摘要

为了为专门的任务创建可部署模型的最可靠方法之一是获得足够数量且高质量的特定任务数据。然而，对于专门的任务，通常这样的数据集并不存在。现有的方法通过从大型语言模型（LLMs）中创建这样的数据，然后将这些知识提炼到较小的模型中来解决这个问题。然而，这些方法受限于LLMs输出的质量，并且往往会生成重复或不正确的数据。在这项工作中，我们提出了基于检索的蒸馏（ReBase）方法，该方法首先从丰富的在线来源中检索数据，然后将其转化为领域特定数据。这种方法极大地增强了数据的多样性。此外，ReBase生成了“思维链”推理，并提炼了LLMs的推理能力。我们在4个基准测试上测试了我们的方法，结果显示我们的方法在SQuAD上的性能提高了高达7.8%，在MNLI上提高了1.37%，在BigBench-Hard上提高了1.94%。

English

One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by creating such data from large language models (LLMs) and then distilling such knowledge into smaller models. However, these methods are limited by the quality of the LLMs output, and tend to generate repetitive or incorrect data. In this work, we present Retrieval Based Distillation (ReBase), a method that first retrieves data from rich online sources and then transforms them into domain-specific data. This method greatly enhances data diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills the reasoning capacity of LLMs. We test our method on 4 benchmarks and results show that our method significantly improves performance by up to 7.8% on SQuAD, 1.37% on MNLI, and 1.94% on BigBench-Hard.