基于变分学习的RLVR参数探索

摘要

探索长期以来一直是强化学习研究的重点。近期，越来越多的证据表明，它也是大语言模型强化学习训练流程中的一个重要组成部分，能够显著影响下游性能。许多现有方法在动作空间层面控制探索，例如使用温度缩放。然而，这些方法无法对词元进行重新排序，只能影响输出分布的方差。这限制了探索能力，并可能导致训练发散或停滞。在本研究中，我们探讨了参数空间探索，即通过从后验分布中采样不同的策略来生成轨迹，每个策略可能探索不同的轨迹。采样多样性较低或较高的策略因此成为探索的互补控制手段。我们提出了一族名为扰动参数策略优化（3PO）的方法，该方法使用不同的采样策略和不同的轨迹分组方式进行奖励估计。在OLMo-3-1025-7B和Qwen2.5-Math-7B上进行的数学推理和代码生成任务实验表明，这些方法在几乎相同的FLOPs成本下，持续优于标准GRPO的平均下游性能。此外，与GRPO和动作空间基线相比，使用多个参数样本在训练过程中始终产生更少的零优势组以及格式错误或错误的轨迹。总体而言，我们的工作为参数空间探索能够改进大语言模型的强化学习提供了证据。

English

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.