ShortOPD：通過短至長同策略蒸餾恢復修剪過的大型語言模型

摘要

結構化剪枝是一種硬體友好的大型語言模型壓縮方法，但過去驗證多集中在選擇題型辨識任務，而相同的壓縮後檢查點在實際部署所需的自由生成任務中卻可能崩潰。兩個觀察揭示此落差：第一，貪婪式 pass@1 在壓縮後幾乎消失，但 pass@k 透過重複取樣能大幅恢復：有用的生成內容僅被降級而非完全抹除。第二，可恢復區間的主要失敗模式源於後綴重複。因此，恢復訓練應使用壓縮模型自身的在策略狀態，並搭配密集的詞元層級監督——這正是「在策略蒸餾」（OPD）的做法：將壓縮前的模型作為凍結教師重複使用。然而，長序列的在策略滾動會將早期恢復預算浪費在低資訊重複後綴上，延緩損失下降。為減緩此浪費，我們提出 \shortopd（短到長在策略蒸餾排程），該方法可偵測經教師確認的重複後綴，將倖存前綴視為每次滾動的有效長度，並將後續滾動預算分配給當下策略能處理的有效長度。在數學、程式碼及開放式生成任務中，\shortopd 將壓縮模型的分數提升至未恢復時約 9 倍，並達到標準恢復方法（不含知識蒸餾的監督微調、知識蒸餾及序列層級知識蒸餾）的 1.6 至 4.4 倍；同時以固定 8192 詞元滾動視窗達成的結果在兩個百分點內匹配，但僅需四分之一訓練時間（8.5 小時對 35.9 小時）及減少 71% 的滾動詞元數量。我們希望此方案能協助將結構化剪枝推向超越困惑度與選擇題基準的邊際效益，更接近可部署的生成品質。

English

Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires. Two observations trace this gap. First, greedy pass@1 nearly vanishes after compression, yet pass@k recovers substantially under repeated sampling: useful generations are demoted, not erased. Second, the recoverable regime fails mainly through suffix repetition. Recovery should therefore train on the compressed model's own on-policy states with dense token-level supervision, which On-Policy Distillation (OPD) provides by reusing the pre-compression model as a frozen teacher. However, long on-policy rollouts spend early recovery budget on low-information repetitive suffixes, delaying loss descent. To mitigate this waste, we propose \shortopd, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes, treats the surviving prefix as each rollout's effective length, and allocates future rollout budgets to the effective lengths the policy can currently use. Across math, code, and open-ended generation, \shortopd\ raises the compressed model's score to about 9times its unrecovered value and 1.6--4.4times standard recovery recipes (SFT w/o KD, KD, and SeqKD), and it matches a fixed 8192-token rollout horizon within two points using a quarter of the training time (8.5 vs.\ 35.9 hours) and 71% fewer rollout tokens. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, a step closer to deployment-ready generation quality.