CAST：博弈求解器作为大语言模型智能体的回合级教师

摘要

训练大语言模型（LLMs）在长周期游戏中进行行动是迈向通用决策制定的一个前景广阔的步骤，然而，基于可验证奖励的强化学习（RLVR）依赖稀疏的最终奖励，这些奖励难以揭示哪些决策决定了成功。更密集的过程信号可以提供缺失的回合级信用分配，但现有信号源难以同时兼顾低成本和高准确性。我们观察到，游戏求解器状态值的变化能够揭示某个动作是否将状态推向成功。基于这一洞察，我们提出了CAST（来自求解器教师的信用分配），该方法将这些值变化转化为求解器优势，并将其作为回合级信号注入到RLVR中。我们进一步证明，在软最优求解器假设下，最大化求解器优势等价于从求解器进行在线策略蒸馏，且仅需标量值而非教师logits。在Sokoban、Minesweeper和Rush Hour这三个游戏中，CAST在域内和未见难度评估下的每个游戏中都优于所有训练的基线，并在ALFWorld和WebShop上取得了最高的平均零样本性能。我们的代码可在 https://github.com/Wloner0809/CAST 获取。

English

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.