dOPSD：擴散語言模型的在線策略自蒸餾

摘要

擴散大型語言模型（dLLMs）透過反覆對遮罩序列進行去雜訊來生成文本，為自迴歸模型提供了一種並行替代方案，但透過訓練後優化來激發強大的推理能力仍然困難：監督式微調屬於離策略學習且會遭受暴露偏差，而强化學習僅能提供稀疏的序列層級獎勵，且在缺乏可處理序列似然的情況下難以應用。在策略自蒸餾（OPSD）提供了一個有前景的替代方案，使用同一個模型同時作為學生與教師，以提供密集的、詞元層級的在策略監督，但其效果取決於賦予教師特權資訊（PI）——通常是推理時無法取得的實例特定真實參考值——因此學生最終只能蒸餾出一個缺乏特權資訊的弱共識策略，對dLLM推理的提升微乎其微。我們提出了dOPSD，此方法直接從學生自身的去雜訊軌跡中推導出教師的特權，利用同一軌跡中較後、解碼程度較高的步驟來評估遮罩位置，而非使用外部標籤，因此教師的優勢來自模型自身的解碼過程；在Dream和LLaDA上，dOPSD提升了域內數學推理與域外程式碼生成，表現優於監督式與在策略基線方法。

English

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.