使用小語言模型對大型語言模型進行微調的仿真器

An Emulator for Fine-Tuning Large Language Models using Small Language Models

October 19, 2023

作者: Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, Christopher D. Manning

cs.AI

摘要

廣泛使用的語言模型（LMs）通常是通過擴展兩階段訓練流程來構建的：首先是使用非常龐大、多樣的文本數據集進行預訓練階段，然後是使用有針對性的示例或其他所需行為的規範進行微調（有時稱為「對齊」）階段。儘管有人假設知識和技能來自預訓練，而微調主要是過濾這些知識和技能組合，但這種直覺並未得到廣泛測試。為了幫助進行測試，我們引入了一種新技術，用於解耦這兩個階段獲得的知識和技能，從而直接回答這個問題：“如果我們將大型模型在預訓練期間學到的知識與小型模型在微調期間學到的知識結合（或反之亦然），會發生什麼？”利用最近在從人類偏好中學習的發展中衍生出的基於強化學習的框架，我們引入了模擬微調（EFT），這是一種合理且實用的方法，用於從近似（或“模擬”）預訓練和不同規模微調的結果中抽樣。我們對EFT的實驗表明，擴展微調往往有助於改進幫助性，而擴展預訓練則有助於提高事實性。除了解耦規模外，我們還表明EFT使得能夠在測試時調整競爭行為特徵，如幫助性和無害性，而無需額外訓練。最後，模擬微調的一個特殊情況，我們稱之為LM放大，通過將大型預訓練模型與小型微調模型集成，從本質上模擬了對大型預訓練模型進行微調的結果。LM放大一致改善了Llama、Llama-2和Falcon系列中指令遵循模型的幫助性和事實性，而無需額外的超參數或訓練。

English

Widely used language models (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?" Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training.