ReflectRL：通过反思到直接推理从黄金负轨迹中学习

摘要

同策略训练已成为提升大语言模型推理能力的一种强效后训练范式，通常借助来自更强专家模型的黄金轨迹进行增强。然而，当专家在更困难的问题上失败时，现有的轨迹引导方法会失去其主要监督来源，这些失败轨迹通常作为负样本被丢弃。我们认为，此类失败——我们称之为黄金负轨迹——若不被视为供模仿的示范，而是被视为可供反思的有缺陷轨迹，仍能提供有价值的推理信号。我们识别出一种"反思优势"：对于困难问题，对有缺陷轨迹进行反思可能比从头直接求解更容易、更有效。基于此，我们提出ReflectRL，一个轻量级即插即用框架，在同策略训练中从黄金负轨迹中学习。ReflectRL首先利用这些轨迹引发反思推理，然后通过反思到直接策略迁移将习得的推理行为迁移回直接推理。在9个基准测试、4个大语言模型底座和4种同策略训练方法上的实验表明，ReflectRL以极小的开销持续提升了推理性能。

English

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.