dOPSD：面向扩散语言模型的在线策略自蒸馏

摘要

扩散大语言模型（dLLM）通过迭代去噪掩码序列生成文本，提供了自回归模型的并行替代方案，但通过后训练激发强推理能力仍具挑战：监督微调属于离策略方法，存在暴露偏差；而强化学习仅提供稀疏的序列级奖励，且因缺乏易于处理的序列似然而难以应用。同策略自蒸馏（OPSD）提供了一种有前景的替代方案，利用同一模型同时扮演学生和教师角色，提供密集的、词元级别的、同策略监督信号。然而，其有效性依赖于向教师模型提供特权信息（PI）——通常是在推理阶段不可获取的实例级真实标签——导致学生最终蒸馏出弱化的、不含特权信息的一致策略，对dLLM推理能力的提升微乎其微。我们提出dOPSD，该方法转而从学生自身的去噪轨迹中推导教师模型的特权：利用同一轨迹中更后期的解码步骤评估掩码位置，而非依赖外部标签。由此，教师模型的优势直接源于模型自身的解码过程。在Dream和LLaDA上的实验表明，dOPSD同时提升了领域内数学推理与领域外代码生成能力，优于监督方法和同策略基线。

English

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.