SLPO：通过代理策略扩展潜在推理

摘要

基于可验证奖励的强化学习已成为在显式思维链推理器中实现测试时尺度扩展的主流范式。然而，这种扩展路径仍存在较高的计算成本，因为每个中间步骤都必须解码为语言标记。相比之下，潜在推理将中间计算以连续向量形式承载，并已在更短的计算步数内达到甚至超越显式思维链的性能。尽管前景广阔，潜在推理器仍主要受限于模仿学习，而显式思维链已通过基于结果奖励的强化学习超越了模仿阶段。潜在轨迹缺乏可处理的逐步骤似然度，且在固定思考预算下缺乏自适应终止接口，因此结果奖励无法激发潜在测试时尺度扩展。我们提出替代潜在策略优化（SLPO），将结果奖励强化学习引入自回归潜在推理器：通过潜在转移的实证替代策略密度实现轨迹级信用分配，并设计一个经正确性监督的终止头，结果奖励优化将其精炼为变视界策略。在连续与软思考设置下，SLPO提升了并行采样下的Pass@k指标，并将更长的潜在计算分配给更难的实例，同时实现更高的确定性准确率。

English

Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@k under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.