使用小语言模型对大语言模型进行微调的仿真器

An Emulator for Fine-Tuning Large Language Models using Small Language Models

October 19, 2023

作者: Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, Christopher D. Manning

cs.AI

摘要

广泛使用的语言模型（LMs）通常是通过扩大规模的两阶段训练流程构建的：一个是使用非常庞大、多样化的文本数据集进行预训练阶段，另一个是使用有针对性的示例或其他所需行为规范的微调（有时称为“对齐”）阶段。虽然有人假设知识和技能来自预训练阶段，而微调主要是过滤这些知识和技能集，但这种直觉尚未得到广泛测试。为了帮助进行测试，我们引入了一种新颖的技术，用于解耦这两个阶段获得的知识和技能，从而直接回答一个问题：“如果我们将大模型在预训练期间学到的知识与小模型在微调期间学到的知识（或反之亦然）结合，会发生什么？”利用最近在从人类偏好中学习方面的发展中提出的基于RL的框架，我们引入了模拟微调（EFT），这是一种原则性和实用的方法，用于从近似（或“模拟”）预训练和微调结果的分布中进行采样。我们使用EFT进行的实验表明，扩大微调往往会提高实用性，而扩大预训练往往会提高事实性。除了解耦规模外，我们展示了EFT可以在测试时调整有竞争关系的行为特征，如实用性和无害性，而无需额外训练。最后，一种特殊情况的模拟微调，我们称之为LM上缩放，通过将大型预训练模型与小型微调模型组合起来，避免了资源密集型的大型预训练模型微调，从本质上模拟了对大型预训练模型进行微调的结果。上缩放一致提高了Llama、Llama-2和Falcon系列指令遵循模型的实用性和事实性，而无需额外的超参数或训练。

English

Widely used language models (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?" Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training.