切勿丢弃你的预训练模型

摘要

对齐训练存在权衡：它帮助语言模型（LMs）提升推理和指令遵循能力，但可能在创造力和校准等技能上有所欠缺，而这些正是未对齐的基础模型更擅长的。我们旨在通过模型协作实现两全其美，让训练管道中的不同模型相互协作、优势互补。鉴于LM响应中交织着适合不同模型的技能，我们提出了切换生成（Switch Generation），让预训练和对齐的模型版本在响应序列中轮流“发言”。具体而言，我们通过从多样查询和上下文中选择不同模型生成下一段的结果中学习，训练一个切换器LM。在推理时，切换器LM引导不同模型检查点动态生成最需要其优势的下一段。在8个模型协作基线和18个数据集上的广泛实验表明：1）模型协作在18个任务中的16个上持续超越单个模型；2）切换生成平均进一步超越基线12.9%。深入分析揭示，切换生成发现了组合技能，解决了单个模型难以应对的问题，并能泛化到未见过的模型和任务，重新利用和转化昂贵模型训练管道中通常被废弃的副产品。

English

Alignment training has tradeoffs: it helps language models (LMs) gain in reasoning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models in the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose Switch Generation, where pretrained and aligned model versions take turns to ``speak'' in a response sequence. Specifically, we train a switcher LM by learning from outcomes of choosing different models to generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints to dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18 datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch Generation further outperforms baselines by 12.9% on average. Further analysis reveals that Switch Generation discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by-products in expensive model training pipelines that are otherwise discarded.

切勿丢弃你的预训练模型

Don't Throw Away Your Pretrained Model

摘要

Support