Self-OPD：无教师流匹配模型的同策略蒸馏

摘要

同策略蒸馏（OPD）利用预训练的任务特定教师模型提供密集监督信号，已在大语言模型（LLMs）中取得显著成功，并最近被适配用于流匹配模型。然而，该范式存在两大问题：首先，为每个新目标单独训练一个任务特定的教师模型会带来高昂的计算成本；其次，教师与学生分布之间的差异往往导致沿生成轨迹的误差累积。在本文中，我们提出Self-OPD，一种用于流匹配模型的无教师OPD框架，它将学生自身的自我探索转化为逐步监督。在每个时间步，Self-OPD将确定性下一状态预测分支为K个随机SDE候选，利用ODE采样器进行展开，并将它们的奖励与确定性自参考基线比较，得到归一化优势。速度场通过一个全分支拉推目标进行优化，其中高优势分支吸引学生，低优势分支在方向感知衰减和SDE方差归一化下排斥学生。对于多目标对齐，Self-OPD在奖励层面融合归一化分数，避免了直接的梯度冲突。在单一与混合奖励基准上的实验表明，Self-OPD在无需任务特定教师的情况下优于先前的RL和OPD方法。

English

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce Self-OPD, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into K stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.

Self-OPD：无教师流匹配模型的同策略蒸馏

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

摘要

Support