TTPO：测试时策略优化

摘要

近期突出的训练后方法，如强化学习（RL）和在线策略自蒸馏（OPSD），已推动大语言模型在数学推理方面取得快速进展，然而它们对真实标签的依赖阻碍了测试时训练（TTT）。用多数投票伪标签替代真实标签是一种自然的替代方案，但该方法存在脆弱性：一次错误的投票会污染教师模型并误导每一个词元。我们观察到这种失败模式具有不对称性：无论投票本身是否正确，与伪标签不一致的采样轨迹通常是错误的。基于这一观察，我们提出了测试时策略优化（TTPO），这是一种不对称的目标函数，通过OPSD蒸馏一致的采样轨迹，并通过分组RL惩罚不一致的采样轨迹。词元级选择进一步优化了两个分支：蒸馏降低已收敛位置的权重，而RL仅惩罚高置信度的错误。即使在伪标签频繁出错的情况下，两种更新仍保持良好有效性，并且随着模型改进，多数投票路由能产生更紧密的自监督信号。在无需任何标签的情况下，TTPO在五个竞赛级基准上达到了与标签监督OPSD相当的性能，在TTT中将Qwen3-1.7B从38.0%提升至45.2%，在无思考模式下取得了+25.2%至+36.4%的提升，并展现出强大的跨任务泛化能力。

English

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.

TTPO：测试时策略优化

TTPO: Test-Time Policy Optimization

摘要

Support