TREK：蒸餾以探索，強化以精煉

摘要

群體相對策略最佳化（Group Relative Policy Optimization, GRPO）在當前策略已能對有用推理軌跡進行取樣時效果良好，但面對正確解模式位於學生在策略支持範圍之外的困難提示時，則會陷入停滯。我們提出 TREK（教師引導探索透過前向KL），這是一種簡單的分階段程序，其利用蒸餾的目的並非模仿，而是擴展探索支持範圍。TREK 的關鍵優勢在於其通用性：由於它僅需使用已驗證的輸出軌跡，因此可採用外部黑盒教師、白盒教師，或在額外推論上下文下使用相同模型；即使無法取得教師內部資訊，也能有效辨識哪些困難提示樣本最值得鞏固。TREK 首先找出未受輔助學生通過率極低的提示，向提案來源查詢以產生經驗證的候選解，保留由當前學生似然排序的前 r 個提案，進行短暫的前向KL階段以將這些已驗證的解模式拉入學生的支持範圍，然後回到標準的在策略 GRPO 微調。在數學推理方面，使用 DeepSeek-V4 提案的 TREK 在 AIME 2024 與 AIME 2025 的所有測試規模上均提升了 Qwen3 模型表現；對於 Qwen3-8B，它在 AIME 2025 上從 36.9 提升至 40.3，在 AIME 2024 上從 47.9 提升至 51.1（avg@16），而自我上下文變體在無外部教師的情況下分別達到 38.5 與 49.6。在代理任務上，TREK 將 ALFWorld 的成功率從 75.8 提升至 82.8，ScienceWorld 的成功率從 12.5 提升至 26.7；值得注意的是，在最困難的任務類型中，TREK 在訓練初期即達到高成功率，而未受輔助的 GRPO 則需要顯著更多最佳化步驟才能達到相當水準。

English

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-r proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.