再思考大语言模型的在策略蒸馏 II：单个训练示例

摘要

在线策略蒸馏（OPD）结合了学生模型生成的轨迹展开与教师模型提供的密集词元级监督。已有工作主要研究其算法行为，而训练数据的作用尚不明确。我们通过在单条查询上训练，在数据极小极限下考察了这一作用。单次查询的OPD在数百步内持续改进，并在多个任务领域和模型家族中恢复了全数据OPD的大部分收益。我们通过训练过程中访问的状态以及学生与教师对齐的速率来解释这一结果。我们度量了状态覆盖率，即全数据OPD访问的状态中，某一查询集的轨迹展开所能到达的比例。单条查询即可到达\(71.5\%\)的状态，其中大部分在前100步内完成。添加语义不同的查询会同时提升覆盖率与验证准确率，直至16条查询达到\(98.9\%\)的覆盖率并与全数据训练持平。然而，无论OPD在单条查询还是整个数据集上训练，对齐速度都以相似的速率放缓，即使状态集合固定不变，也需要数百步才能完成吸收。因此，OPD面临的问题在于数据过剩而算法饥渴——其轨迹展开能迅速暴露广泛范围的监督信号，而学生吸收这些监督的速度却越来越慢。状态覆盖率的结果可推广至多教师OPD，每个领域16条语义多样的查询即可与全数据MOPD持平。作为进一步的压力测试，内容精简的模板和领域外的WildChat查询也接近真实查询基线的表现。任务内容与由此诱导的状态覆盖率因此可以相互分离。我们希望这些发现能将未来工作引向OPD的步数效率问题，并促使学界重新审视前沿模型后期训练中其近期成功所依赖的数据与机制。

English

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.