TREK：蒸馏以探索，强化以精炼

摘要

群体相对策略优化（GRPO）在当前策略已能采样到有效推理轨迹时表现良好，但对于正确解答模式位于学生当前策略支持范围之外的困难提示，其优化会陷入停滞。我们提出TREK（基于正向KL散度的教师引导探索法），这是一种简单的分阶段流程，利用蒸馏技术的目的并非模仿，而是扩展探索支持范围。TREK的关键优势在于其通用性：由于仅需使用已验证的输出轨迹，它既可以借助外部黑盒教师、白盒教师，也可以利用同一模型在额外推理上下文下的自身能力；即便无法获取教师内部机制，它也能高效识别哪些困难提示样本最值得巩固。TREK首先识别学生未经辅助时通过率极低的提示，向提议源查询以生成经过验证的候选解答，保留按当前学生似然度排序的前r个候选方案，执行短期正向KL散度阶段将这些已验证的解答模式纳入学生支持范围，随后回归标准的在线策略GRPO精炼阶段。在数学推理任务中，采用DeepSeek-V4提议的TREK方法在AIME 2024和AIME 2025数据集上对所有测试规模的Qwen3模型均有提升；对于Qwen3-8B模型，在AIME 2025上的成绩从36.9提升至40.3，在AIME 2024上从47.9提升至51.1（avg@16）。而自上下文变体在无需外部教师的情况下分别达到38.5和49.6。在智能体任务中，TREK将ALFWorld的成功率从75.8提升至82.8，ScienceWorld的成功率从12.5提升至26.7；值得注意的是，在最困难的任务类型中，TREK在训练初期即达到较高成功率，而未经辅助的GRPO需要更多优化步骤才能达到同等水平。

English

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-r proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.