CEPO: 基于对比证据策略优化的RLVR自蒸馏

摘要

在基于可验证奖励的强化学习（RLVR）中，当模型生成正确解答时，每个token获得相同的奖励信号，无论该token属于关键推理步骤还是语法填充词。一种自然的改进方法是利用正确答案作为教师信号来约束模型，识别出若模型提前知道答案时会生成的不同token。然而，先前研究表明该方法存在两种缺陷：要么因将答案泄露至梯度而破坏训练，要么产生无法区分关键步骤与填充词的微弱信号——因为相对于模型基线，两者均表现出同等程度的"意外性"。本文提出对比证据策略优化（Contrastive Evidence Policy Optimization, CEPO），该方法在每个token处提出更精准的问题：不仅关注"正确答案是否偏好该token？"，更追问"该token是否同时满足正确答案偏好且错误答案排斥？"。同时满足两者的token为真实推理步骤，反之则为填充词。错误答案教师信号由训练批次中已有的被拒绝轨迹构建，无需额外采样成本。理论证明，CEPO继承了先前最优方法的所有结构安全性保证，同时在关键token处实现更明确的信用分配，且这种改进在填充词位置完全消失。实验表明，在五个多模态数学推理基准测试中，CEPO在2B和4B规模下分别达到43.43%和60.56%的平均准确率，而GRPO在相同训练预算下仅为41.17%和57.43%。基于分布匹配的自蒸馏方法（OPSD、SDPO）的性能甚至低于未训练基线，实证验证了我们理论预测的信息泄露问题。我们的代码已开源至https://github.com/ahmedheakl/CEPO。

English

When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every token: not just "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?" A token satisfying both is a genuine reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five multimodal mathematical reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.