超越马尔可夫性：通过贝叶斯自适应强化学习实现大语言模型的反思性探索

Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning

May 26, 2025

作者: Shenao Zhang, Yaqing Wang, Yinxiao Liu, Tianqi Liu, Peter Grabowski, Eugene Ie, Zhaoran Wang, Yunxuan Li

cs.AI

摘要

通过强化学习（RL）训练的大型语言模型（LLMs）已展现出强大的推理能力和涌现的反思行为，如回溯与错误修正。然而，传统的马尔可夫强化学习将探索局限于训练阶段，以学习最优确定性策略，并仅通过当前状态依赖历史上下文。因此，尚不清楚反思推理是否会在马尔可夫强化学习训练期间自然涌现，或为何它们在测试时具有优势。为解决这一问题，我们将反思探索重新置于贝叶斯自适应强化学习框架中，该框架明确优化了在马尔可夫决策过程后验分布下的期望回报。这一贝叶斯公式通过信念更新，内在激励了奖励最大化的利用与信息收集的探索。我们提出的算法BARL指导LLM根据观察结果拼接和切换策略，为模型何时及如何进行反思探索提供了原则性指导。在合成任务和数学推理任务上的实证结果表明，BARL在测试时优于标准马尔可夫强化学习方法，以更高的探索效率实现了更优的令牌利用率。我们的代码已发布于https://github.com/shenao-zhang/BARL。

English

Large Language Models (LLMs) trained via Reinforcement Learning (RL) have exhibited strong reasoning capabilities and emergent reflective behaviors, such as backtracking and error correction. However, conventional Markovian RL confines exploration to the training phase to learn an optimal deterministic policy and depends on the history contexts only through the current state. Therefore, it remains unclear whether reflective reasoning will emerge during Markovian RL training, or why they are beneficial at test time. To remedy this, we recast reflective exploration within the Bayes-Adaptive RL framework, which explicitly optimizes the expected return under a posterior distribution over Markov decision processes. This Bayesian formulation inherently incentivizes both reward-maximizing exploitation and information-gathering exploration via belief updates. Our resulting algorithm, BARL, instructs the LLM to stitch and switch strategies based on the observed outcomes, offering principled guidance on when and how the model should reflectively explore. Empirical results on both synthetic and mathematical reasoning tasks demonstrate that BARL outperforms standard Markovian RL approaches at test time, achieving superior token efficiency with improved exploration effectiveness. Our code is available at https://github.com/shenao-zhang/BARL.