超越马尔可夫性：基于贝叶斯自适应强化学习的大语言模型推理反思探索

Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning

May 26, 2025

作者: Shenao Zhang, Yaqing Wang, Yinxiao Liu, Tianqi Liu, Peter Grabowski, Eugene Ie, Zhaoran Wang, Yunxuan Li

cs.AI

摘要

通过强化学习（RL）训练的大型语言模型（LLMs）展现了强大的推理能力和涌现的反思行为，如回溯和错误纠正。然而，传统的马尔可夫强化学习将探索限制在训练阶段，以学习一个最优的确定性策略，并且仅通过当前状态依赖历史上下文。因此，尚不清楚在马尔可夫强化学习训练期间是否会涌现出反思推理，或者为何这些行为在测试时有益。为解决这一问题，我们在贝叶斯自适应强化学习框架中重新构建了反思探索，该框架明确优化了在马尔可夫决策过程后验分布下的预期回报。这种贝叶斯公式通过信念更新，内在激励了奖励最大化的利用和信息收集的探索。我们提出的算法BARL指导LLM根据观察到的结果拼接和切换策略，为模型何时以及如何进行反思探索提供了原则性指导。在合成任务和数学推理任务上的实证结果表明，BARL在测试时优于标准的马尔可夫强化学习方法，以更高的探索效率实现了更优的标记效率。我们的代码可在https://github.com/shenao-zhang/BARL获取。

English

Large Language Models (LLMs) trained via Reinforcement Learning (RL) have exhibited strong reasoning capabilities and emergent reflective behaviors, such as backtracking and error correction. However, conventional Markovian RL confines exploration to the training phase to learn an optimal deterministic policy and depends on the history contexts only through the current state. Therefore, it remains unclear whether reflective reasoning will emerge during Markovian RL training, or why they are beneficial at test time. To remedy this, we recast reflective exploration within the Bayes-Adaptive RL framework, which explicitly optimizes the expected return under a posterior distribution over Markov decision processes. This Bayesian formulation inherently incentivizes both reward-maximizing exploitation and information-gathering exploration via belief updates. Our resulting algorithm, BARL, instructs the LLM to stitch and switch strategies based on the observed outcomes, offering principled guidance on when and how the model should reflectively explore. Empirical results on both synthetic and mathematical reasoning tasks demonstrate that BARL outperforms standard Markovian RL approaches at test time, achieving superior token efficiency with improved exploration effectiveness. Our code is available at https://github.com/shenao-zhang/BARL.