為何多步驟工具使用的強化學習會崩潰，以及監督訊號如何修復此問題

摘要

工具使用使大型語言模型（LLM）能夠執行複雜任務，而近期以代理為基礎的強化學習（RL）方法展現出提升模型能力的潛力。然而，單獨使用RL往往會導致工具使用任務中的不穩定性或效益有限。在我們的實驗中，部分模型出現災難性崩潰，表現突然下降，且工具調用結構失效。分析顯示，這些失敗源於特定控制標記出現意外的機率尖峰，從而破壞了結構化執行，然而底層的工具使用能力仍然完好，只是被特定格式所掩蓋。為了解決這個問題，我們系統性地研究了多種監督信號，包括離策略監督、基於提示的引導、錯誤範例監督等，並在同步與交錯訓練方案下加以應用。我們發現，將監督式微調（SFT）與RL交錯使用能顯著提升穩定性，但在格式與內容分佈外（OOD）評估中表現下降。我們也分析了學習率的影響以及跨設定的泛化情形。這些結果凸顯了理解RL失敗的重要性，並展示了多樣化的監督信號如何引導探索式學習，從而實現對LLM在複雜多步工具使用任務中的穩健訓練。我們的程式碼可於 https://github.com/hypasd-art/Tool-RL-Box 取得。

English

Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.