TTPO：測試時策略優化

摘要

近期突出的後訓練方法，例如強化學習（RL）與在策略自蒸餾（OPSD），已大幅推動大型語言模型在數學推理上的進展；然而，這些方法依賴真實標籤，因此無法進行測試時訓練（TTT）。以多數投票偽標籤取代真實標籤是自然的替代方案，但其相當脆弱：一次錯誤的投票便會污染教師模型，並誤導所有詞元。我們觀察到這種失敗模式具有不對稱性：與偽標籤不一致的軌跡通常是錯誤的，無論投票本身是否正確。基於此觀察，我們提出測試時策略優化（TTPO），這是一種不對稱目標函數，透過 OPSD 蒸餾一致的軌跡，並以分組 RL 懲罰不一致的軌跡。詞元層級選擇進一步細化這兩個分支：蒸餾會降低已收斂位置的權重，而 RL 僅懲罰高置信度的錯誤。即使偽標籤頻繁出錯，兩種更新依然保持穩健；且隨著模型改進，多數投票路由會產生更精確的自監督。在完全沒有標籤的情況下，TTPO 在五個競賽級基準上與標籤監督的 OPSD 並駕齊驅，將 Qwen3-1.7B 在 TTT 中從 38.0% 提升至 45.2%，在未進行思考時獲得 +25.2% 至 +36.4% 的提升，並展現出強勁的跨任務泛化能力。

English

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.

TTPO：測試時策略優化

TTPO: Test-Time Policy Optimization

摘要

Support