TinyR1-32B-Preview：透過分支合併蒸餾提升準確率

TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation

March 6, 2025

作者: Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Change Jia, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai-er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Bin Cui, Tong Yang, Xiangzheng Zhang

cs.AI

摘要

在保持性能的同时縮減大型語言模型（LLMs）的規模已成為一項備受關注的挑戰。然而，現有的方法，如模型蒸餾和遷移學習，往往難以實現高準確率。為解決這一限制，我們引入了分支合併蒸餾方法，該方法通過兩個階段增強模型壓縮：(1) 分支階段，在此階段，大型教師模型的知識通過領域特定的監督微調（SFT）有選擇地蒸餾到專用學生模型中；(2) 合併階段，在此階段，這些學生模型被合併，以實現跨領域知識轉移並提升泛化能力。我們使用DeepSeek-R1作為教師模型，DeepSeek-R1-Distill-Qwen-32B作為學生模型，驗證了我們的蒸餾方法。最終合併的模型TinyR1-32B-Preview在多個基準測試中均優於其對應的DeepSeek-R1-Distill-Qwen-32B，包括數學（+5.5分）、編碼（+4.4分）和科學（+2.9分），同時在AIME 2024上實現了與DeepSeek-R1近乎相當的性能。分支合併蒸餾方法為創建更小、高性能且計算成本和時間更低的LLMs提供了一種可擴展的解決方案。

English

The challenge of reducing the size of Large Language Models (LLMs) while maintaining their performance has gained significant attention. However, existing methods, such as model distillation and transfer learning, often fail to achieve high accuracy. To address this limitation, we introduce the Branch-Merge distillation approach, which enhances model compression through two phases: (1) the Branch Phase, where knowledge from a large teacher model is selectively distilled into specialized student models via domain-specific supervised fine-tuning (SFT); And (2) the Merge Phase, where these student models are merged to enable cross-domain knowledge transfer and improve generalization. We validate our distillation approach using DeepSeek-R1 as the teacher and DeepSeek-R1-Distill-Qwen-32B as the student. The resulting merged model, TinyR1-32B-Preview, outperforms its counterpart DeepSeek-R1-Distill-Qwen-32B across multiple benchmarks, including Mathematics (+5.5 points), Coding (+4.4 points) and Science (+2.9 points), while achieving near-equal performance to DeepSeek-R1 on AIME 2024. The Branch-Merge distillation approach provides a scalable solution for creating smaller, high-performing LLMs with reduced computational cost and time.