切勿輕易捨棄您的預訓練模型

摘要

對齊訓練存在權衡：它幫助語言模型（LMs）提升推理和指令遵循能力，但可能在創造力和校準等技能上有所損失，這些方面未對齊的基礎模型表現更佳。我們旨在通過模型協作來兼顧兩者之長，讓訓練流程中的不同模型相互協作、互補。由於LM回應中交織著適合不同模型的技能，我們提出了切換生成（Switch Generation），讓預訓練和對齊的模型版本在回應序列中輪流「發言」。具體而言，我們通過學習在不同查詢和上下文中選擇不同模型生成下一段的結果，來訓練一個切換器LM。在推理時，切換器LM引導不同的模型檢查點動態生成下一段，在它們最擅長的領域發揮作用。通過與8個模型協作基線和18個數據集的廣泛實驗表明：1）模型協作在18項任務中的16項上持續超越單一模型，2）切換生成進一步平均超越基線12.9%。深入分析揭示，切換生成發現了組合技能來解決單一模型難以應對的問題，並能泛化到未見的模型和任務，重新利用和轉化昂貴模型訓練流程中通常被丟棄的副產品。

English

Alignment training has tradeoffs: it helps language models (LMs) gain in reasoning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models in the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose Switch Generation, where pretrained and aligned model versions take turns to ``speak'' in a response sequence. Specifically, we train a switcher LM by learning from outcomes of choosing different models to generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints to dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18 datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch Generation further outperforms baselines by 12.9% on average. Further analysis reveals that Switch Generation discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by-products in expensive model training pipelines that are otherwise discarded.

切勿輕易捨棄您的預訓練模型

Don't Throw Away Your Pretrained Model

摘要

Support