β-OPSD：以策略优化推导，以自蒸馏训练

摘要

在线策略自蒸馏（OPSD）是改进推理语言模型的一种有前景的方法，但在实践中仍显脆弱：要使其可靠运行，往往需要大量的工程投入。我们识别出这一困难的根源：原始OPSD正是更广泛策略优化族中β=1的特例，其中β对锚定学生模型至参考策略的KL惩罚进行加权。这一等价关系将β从隐含固定为1的取值转变为可控的正则化参数，从而产生一种更通用的公式化表达，在接近参考策略与特权教师引导之间进行权衡。我们提出β-OPSD，并将其最优策略推导为参考策略与特权教师之间的几何插值。然而，直接使用强化学习优化该目标将代价高昂且方差较大。我们不直接优化强化学习目标，而是将其闭式解转化为蒸馏目标。每个β值沿参考策略到教师的路径选取一个目标，我们通过混合二者的词元级logits来高效实现。如此，低成本的蒸馏即可逼近高代价策略优化的解。回报归因（return-to-go）的信用分配进一步使词元更新与序列级目标对齐，同时保留OPSD的简洁性。在数学推理基准上的实验表明，β-OPSD持续优于原始OPSD，提升了优化稳定性与下游推理性能。我们的结果为从自蒸馏到策略优化再返回提供了一条原则性路径，且不牺牲使OPSD具备实用性的效率。

English

On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the β=1 member of a broader policy-optimization family, where β weights the KL penalty anchoring the student to a reference policy. This equivalence turns β from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce β-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of β selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that β-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.