多世界择优：通过max@k优化实现强化学习与最佳N样本采样的对齐

摘要

可验证奖励强化学习（RLVR）在数学与编程领域的应用已显著提升大语言模型的推理与问题解决能力。尽管该技术在单次生成问题求解中表现成功，但强化学习微调过程可能削弱模型的探索能力，具体表现为生成多样性的下降，进而导致在大N值的最佳N采样（Best-of-N）中性能退化。本研究聚焦于优化max@k指标——该指标是pass@k的连续泛化形式。我们推导出用于直接优化该指标的无偏同策略梯度估计，并将推导延伸至现代RLVR算法中常见的异策略更新机制，以提升样本效率。实验表明，我们的目标函数能有效优化异策略场景下的max@k指标，使模型与最佳N推理策略保持一致。

English

The application of Reinforcement Learning with Verifiable Rewards (RLVR) to mathematical and coding domains has demonstrated significant improvements in the reasoning and problem-solving abilities of Large Language Models. Despite its success in single generation problem solving, the reinforcement learning fine-tuning process may harm the model's exploration ability, as reflected in decreased diversity of generations and a resulting degradation of performance during Best-of-N sampling for large N values. In this work, we focus on optimizing the max@k metric, a continuous generalization of pass@k. We derive an unbiased on-policy gradient estimate for direct optimization of this metric. Furthermore, we extend our derivations to the off-policy updates, a common element in modern RLVR algorithms, that allows better sample efficiency. Empirically, we show that our objective effectively optimizes max@k metric in off-policy scenarios, aligning the model with the Best-of-N inference strategy.

多世界择优：通过max@k优化实现强化学习与最佳N样本采样的对齐

The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation

摘要

Support