一幣勝千幣：基於低秩克隆的高效知識蒸餾

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

May 19, 2025

作者: Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu

cs.AI

摘要

訓練高性能的小型語言模型（SLMs）成本依然高昂，即便採用知識蒸餾和從大型教師模型中剪枝的方法。現有研究常面臨三大挑戰：（1）硬剪枝導致的信息損失，（2）表示對齊效率低下，以及（3）信息性激活，尤其是前饋網絡（FFNs）激活的利用不足。為應對這些挑戰，我們提出了低秩克隆（Low-Rank Clone, LRC），這是一種高效的預訓練方法，旨在構建行為上與強力教師模型等效的SLMs。LRC通過訓練一組低秩投影矩陣，實現了軟剪枝——通過壓縮教師模型權重，以及激活克隆——通過對齊學生模型與教師模型的激活，包括FFN信號。這一統一設計在最大化知識轉移的同時，省去了顯式對齊模塊的需求。利用開源教師模型（如Llama-3.2-3B-Instruct、Qwen2.5-3B/7B-Instruct）進行的廣泛實驗表明，LRC在僅使用200億令牌的情況下，匹配甚至超越了基於數萬億令牌訓練的最新模型，實現了超過1000倍的訓練效率。我們的代碼和模型檢查點可於https://github.com/CURRENTF/LowRankClone 和 https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf 獲取。

English

Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.