基於變分學習的可驗證獎勵強化學習參數探索

摘要

長期以來，探索一直是強化學習研究的核心主題。近期，越來越多的證據顯示，探索也是大型語言模型（LLM）強化學習訓練流程中的重要組成部分，可對下游表現產生顯著影響。許多現有方法在動作空間中控制探索，例如透過溫度縮放。然而，這類方法無法重新排列詞元，僅能影響輸出分佈的變異數，因而限制了探索的範圍，可能導致訓練發散或停滯。在本研究中，我們探討參數空間探索——透過從後驗分佈中採樣不同的策略來生成軌跡，而每個策略皆可能探索不同的軌跡。因此，採樣多樣性較低或較高的策略，成為一種互補的探索控制手段。我們提出一系列稱為「擾動參數策略最佳化」（Perturbed Parameter Policy Optimization，簡稱3PO）的方法，這些方法採用不同的採樣策略與不同的軌跡分組方式來估計獎勵。在OLMo-3-1025-7B與Qwen2.5-Math-7B上針對數學推理與程式碼生成任務進行的實驗顯示，與標準GRPO相比，這些方法在近乎相同的FLOPs成本下，一致地改善了平均下游表現。此外，與GRPO及動作空間基線相比，使用多個參數樣本在訓練過程中所產生的零優勢群組，以及格式錯誤或不正確的軌跡數量也更少。整體而言，我們的研究為參數空間探索能改善大型語言模型的強化學習提供了實證支持。

English

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.