CAST：遊戲求解器作為LLM代理的回合級教師

摘要

訓練大型語言模型（LLM）在長時域遊戲中行動，是邁向通用型決策制定的一個有前途的步驟，然而，基於可驗證獎勵的強化學習（RLVR）依賴於稀疏的最終獎勵，這種獎勵幾乎無法揭示哪些決策決定了成功。更密集的過程信號可以提供這種缺失的回合級信用分配，但現有的來源很難同時保持低成本和高準確性。我們觀察到，遊戲求解器的狀態值變化揭示了某個動作是否使狀態朝成功方向前進。基於這一見解，我們提出了CAST（來自求解器教師的信用分配），它將這些狀態值變化轉換為求解器優勢，並將其作為回合級信號注入RLVR。我們進一步證明，在軟最優求解器假設下，最大化求解器優勢等價於從求解器進行在線蒸餾，僅需標量值而非教師的logits。在倉庫迷、踩地雷和交通堵塞等遊戲中，CAST在領域內及未見過的難度評估下，於每個遊戲中均優於所有訓練基線，並在ALFWorld和WebShop上達到最高的平均零樣本性能。我們的程式碼可在 https://github.com/Wloner0809/CAST 取得。

English

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.