Self-OPD：無教師流匹配模型之在策略蒸餾

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

August 27, 2026

作者: Shiyi Zhang, Mushui Liu, Yunze Tong, Wanggui He, Siyu Zou, Jinlong Liu, Yunlong Yu, Jian Song, Hao Jiang, Pipei Huang, Bo Zheng

cs.AI

摘要

同策略蒸餾（On-policy distillation, OPD）利用預訓練的專業教師模型提供密集監督信號，已在大型語言模型（LLMs）中取得顯著成功，並於近期被應用於流匹配模型。然而，此範式存在兩大問題：首先，為每個新目標訓練一個獨立且任務特定的教師模型會造成高昂的計算成本；其次，教師與學生分佈之間的差異常導致生成軌跡上的累積誤差。本文提出 Self-OPD，一種針對流匹配模型的無教師 OPD 框架，將學生自身的自我探索轉化為逐步監督。在每個時間步，Self-OPD 將確定性的下一狀態預測分支為 K 個隨機 SDE 候選，使用 ODE 採樣器對其進行展開，並將其獎勵與確定性自參照基準比較，以獲得歸一化優勢。速度場透過全分支推拉目標進行優化，其中在方向感知衰減與 SDE 方差歸一化下，高優勢分支吸引學生，低優勢分支排斥學生。在多目標對齊方面，Self-OPD 在獎勵層級融合歸一化分數，避免直接的梯度衝突。在單一與混合獎勵基準上的實驗表明，Self-OPD 在無需任務特定教師的情況下優於先前的 RL 和 OPD 方法。

English

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce Self-OPD, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into K stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.