透過自蒸餾增強基於評分標準的強化學習

摘要

基於評分標準的強化學習（rubric-based RL）近期在改善大型語言模型（LLM）於開放式任務的表現上展現出潛力。其一個廣為人知的限制在於探索有限：沒有任何生成軌跡能滿足的標準（即未探索標準，Unexplored Criteria, UC）無法獲得任何最佳化訊號。近期方法透過在生成軌跡期間將評分標準資訊納入外部引導來解決此問題，但它們引入了訓練與推論不一致的現象：策略是在外部引導下產生的軌跡上進行最佳化，而在推論時卻沒有這種引導，導致自迴歸解碼過程中誤差累積。此外，這些以探索為重點的方法忽略了我們稱之為受抑制標準（Suppressed Criteria, SC）的根本不同失敗模式：某些標準雖有部分軌跡能滿足，但其學習訊號在最佳化過程中遺失，原因是標量獎勵聚合將它們的聚合優勢分配為非正值。我們的分析顯示，SC 非常普遍：在整個訓練過程中，超過 57% 的樣本出現此失敗模式，平均每個樣本有 1.8 個 SC。為了同時解決 UC 和 SC，且不引入訓練與推論不一致的問題，我們提出了標準蒸餾策略最佳化（Criterion-Distilled Policy Optimization, CriPO），透過在策略自蒸餾（on-policy self-distillation）來增強基於評分標準的強化學習。針對 UC，CriPO 建構了一個標準注入自教師（criterion-injection self-teacher），並計算局部化的前向 KL 散度損失，將缺失的行為注入策略中。針對 SC，CriPO 採用反事實自教師（counterfactual self-teacher），在負優勢的生成軌跡中定位與標準相關的 token，並將其 token 層級的優勢翻轉為正值，從而保留否則會被抑制的有用模式。在醫學和科學基準上的實驗表明，CriPO 持續優於基於評分標準的強化學習，能以約 2 倍更少的最佳化步驟達成更強的最終表現。

English

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately 2times fewer optimization steps.