通过自蒸馏提升基于量规的强化学习

摘要

基于评分标准的强化学习近期在提升大语言模型处理开放式任务方面展现出良好前景。该范式一个广为人知的局限性在于探索不足：任何轨迹都未能满足的标准（Unexplored Criteria, UC）无法获得优化信号。近期方法通过在轨迹生成过程中引入评分标准信息作为外部引导来解决这一问题，但由此引入了训练-推理不一致：策略是在外部引导下生成的轨迹上进行优化的，而在推理阶段该引导并不存在，从而导致自回归解码过程中的误差累积。此外，这些以探索为导向的方法忽视了一种根本不同的失败模式，我们将其称为被抑制标准（Suppressed Criteria, SC）——即某些轨迹满足该标准，但其学习信号在优化过程中丢失，因为标量奖励聚合赋予了它们非正的聚合优势。我们的分析表明，SC现象极为普遍：在整个训练过程中，超过57%的样本表现出这种失败模式，平均每个样本存在1.8个SC。为了在不引入训练-推理不一致的前提下同时解决UC和SC问题，我们提出了标准蒸馏策略优化（Criterion-Distilled Policy Optimization, CriPO），该方法通过在线策略自蒸馏来增强基于评分标准的强化学习。针对UC，CriPO构建了一个标准注入式自教师模型，并计算局部前向KL散度损失，以将缺失行为注入策略。针对SC，CriPO采用反事实自教师模型，在负优势轨迹中定位与标准相关的词元，并将其词元级优势翻转为正值，从而保留原本会被抑制的有用模式。在医学和科学基准上的实验表明，CriPO始终优于基于评分标准的强化学习，并且仅需约二分之一的优化步数即可实现更强的最终性能。

English

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately 2times fewer optimization steps.