CAST：面向训练可靠长时程工具调用智能体的批评感知监督方法

摘要

大型语言模型（LLM）智能体越来越多地被部署在长程、交互式且有状态的环境中。在这些场景中，单个错误动作（例如退款了错误的购买）可能导致不可逆的任务失败，必须在其执行前予以拦截。此类失败可能不会在每一次单独运行中出现，但会在多次试验中浮现，因此跨步骤和跨试验的可靠性至关重要。然而，确保智能体的可靠性颇具挑战：即使是前沿大语言模型，也难以为动作可能出错的原因作出解释，尤其是在由领域特定策略支配的冗长且相互交织的轨迹中。近期的许多工作依赖于基于提示的批判智能体，而基于优化的方法缺乏系统化的方式来生成丰富的验证理由以用于训练。我们通过CAST填补了这一空白，这是一个批判感知训练框架，将稀疏的任务结果转化为动作级监督，用于批判学习和策略优化。CAST分析智能体轨迹，以生成结构化的理由，在部分可观测性下解释动作的有效性。由此得到的批判模型被用于构建批判感知的训练数据，以优化策略模型。在动态工具调用基准上微调Qwen3系列模型，CAST提升了跨领域的可靠性，在Retail任务上的pass^4指标上超过GPT-OSS-120B逾10%，并在域外设置的Telehealth任务上额外取得了9%的提升。这些结果表明，批判感知训练提升了LLM智能体在现实动态环境中的鲁棒性。

English

Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.