FuseChat-3.0：偏好优化与异构模型融合的完美结合

FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion

March 6, 2025

作者: Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, Xiaojun Quan

cs.AI

摘要

我們推出FuseChat-3.0，這是一套大型語言模型（LLMs），通過將異構源LLMs的優勢整合到更緊湊的目標LLMs中開發而成。我們的源模型包括強大的Gemma-2-27B-it、Mistral-Large-Instruct-2407、Qwen-2.5-72B-Instruct和Llama-3.1-70B-Instruct。對於目標模型，我們專注於三種廣泛使用的較小變體——Llama-3.1-8B-Instruct、Gemma-2-9B-it和Qwen-2.5-7B-Instruct——以及兩種超緊湊選項，Llama-3.2-3B-Instruct和Llama-3.2-1B-Instruct。為了充分利用這些源模型的多樣化能力，我們開發了一種專為各種任務和領域量身定制的數據構建協議。FuseChat-3.0的訓練管道包含兩個關鍵階段：（1）監督微調（SFT）以對齊目標和源模型的分佈，（2）直接偏好優化（DPO）以應用來自多個源LLMs的偏好來微調目標模型。最終的FuseChat-3.0模型在指令遵循、通用知識、數學和編碼等任務上表現出顯著的性能提升。如圖1所示，使用Llama-3.1-8B-Instruct作為目標模型，我們的融合方法在14個基準測試中平均提升了6.8分。此外，在指令遵循基準測試AlpacaEval-2和Arena-Hard上分別取得了37.1分和30.1分的顯著提升。我們的代碼、模型和數據集可在https://github.com/SLIT-AI/FuseChat-3.0獲取。

English

We introduce FuseChat-3.0, a suite of large language models (LLMs) developed by integrating the strengths of heterogeneous source LLMs into more compact target LLMs. Our source models include the powerful Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For target models, we focus on three widely-used smaller variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along with two ultra-compact options, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source models, we develop a specialized data construction protocol tailored to various tasks and domains. The FuseChat-3.0 training pipeline consists of two key stages: (1) supervised fine-tuning (SFT) to align the target and source model distributions, and (2) Direct Preference Optimization (DPO) to apply preferences from multiple source LLMs to fine-tune the target model. The resulting FuseChat-3.0 models exhibit significant performance gains across tasks such as instruction following, general knowledge, mathematics, and coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target model, our fusion approach achieves an average improvement of 6.8 points across 14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and 30.1 points on the instruction-following benchmarks AlpacaEval-2 and Arena-Hard, respectively. Our code, models, and datasets are available at https://github.com/SLIT-AI/FuseChat-3.0.