AsyncOPD：同策略蒸馏能老旧到何种程度？

摘要

在策略蒸餾（OPD）透過教師反饋指導學生在自己執行的軌跡上進行訓練，正逐漸成為大型語言模型（LLM）後訓練中的關鍵技術。然而，如同強化學習（RL），OPD面臨在策略系統的瓶頸——對於推理任務，軌跡生成可能占據大部分訓練時間。非同步訓練管線可透過將軌跡生成與學習器更新解耦來緩解此瓶頸，但這樣做會引入陳舊策略資料。雖然先前研究已探討過非同步RL中的陳舊資料問題，但其在OPD中的影響仍未得到充分探索。我們首次針對非同步OPD中的陳舊性問題進行系統性研究，專注於一個實務場景：教師反饋以局部KL散度損失實現，且完整詞彙的教師logits因儲存或傳輸成本過高而需採用有限的教師分數快取。首先，我們證明KL散度方向會改變陳舊資料問題的性質：教師加權的前向KL散度對陳舊軌跡更具穩健性，而學生加權的反向KL散度則較為脆弱。其次，針對此脆弱的反向KL案例，我們探討了為穩定非同步RL所設計的方法是否能減輕OPD的陳舊性問題。在我們的實驗中，這些方法並未優於一個更簡單的OPD特定替代方案——在學習器端根據當前學生重新計算反向KL信號。第三，我們分析了有限的教師分數快取如何為稀疏與抽樣反向KL OPD估計器引入偏差-變異權衡。這促使我們採用多樣本蒙地卡羅（MC）方法，其能在降低單樣本方差的同時保留MC的可修正性。最後，我們提出並開源了AsyncOPD——一個基於上述估計器選擇所建構的完全非同步OPD訓練管線。實驗結果顯示，與嚴格同步訓練相比，AsyncOPD將訓練吞吐量提升1.6倍至3.8倍，同時達到可比的準確度。

English

On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6times to 3.8times over strict synchronous training while reaching comparable accuracy.