CAST：用於訓練可靠長程工具呼叫代理的批判感知監督

摘要

大型語言模型（LLM）智能體日益被部署於長時程、互動式且具狀態的環境中。在這些設定下，單一錯誤動作——例如退款錯誤的購買項目——可能導致不可逆的任務失敗，且必須在執行前予以攔截。這類失敗未必出現在每一次單獨運行中，但可能在重複試驗中浮現，使得跨步驟與跨試驗的可靠性至關重要。然而，確保智能體的可靠性極具挑戰：即便是前沿的大型語言模型，也難以解釋某個動作何以可能出錯，尤其是在受領域特定規則約束的冗長交織軌跡中。近期許多研究依賴基於提示詞的批判智能體，而基於優化的方法則缺乏系統性的方式來產生豐富的驗證理據以供訓練之用。我們透過 CAST 填補此一缺口，這是一個批判感知訓練框架，可將稀疏的任務結果轉化為動作層級的監督訊號，用於批判學習與策略優化。CAST 分析智能體軌跡，以綜合出具結構化的理據，解釋在部分可觀測性下動作的有效性。所產生的批判模型被用於建構批判感知訓練資料，以優化策略模型。在動態工具呼叫基準上微調 Qwen3 系列模型後，CAST 提升了跨領域的可靠性，在零售任務中較 GPT-OSS-120B 高出逾 10% 的 pass^4 表現，並在域外設定的遠距醫療任務中額外獲得 9% 的提升。這些結果證明，批判感知訓練能提升大型語言模型智能體在真實動態環境中的穩健性。

English

Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.