从成功之流中追溯能动性失败

摘要

面向基于大语言模型的代理系统的故障归因（即识别故障轨迹中导致任务失败的步骤）对于调试和改进此类系统至关重要。现有方法要么依赖基于提示的流水线（计算成本高昂），要么需要对带有步骤级错误标注的故障轨迹进行后训练（标注成本高且难以扩展）。本文认为，实用的故障归因模型应具备轻量级特性，且无需基于故障数据的步骤级监督即可训练。为此，我们研究无监督故障归因问题：仅使用成功轨迹进行训练，在推理阶段识别故障轨迹中的错误步骤。我们提出OAT方法，将问题转化为基于神经控制微分方程的单类学习，在潜在空间中建模成功轨迹的动态模式。推理时，故障轨迹中每个步骤根据其与成功轨迹动态模式的偏离程度获得异常分数，并据此构建错误步骤集合。实验表明，仅需100条成功轨迹进行训练，OAT比基于提示的基线方法快200-5000倍，同时在域内数据集和域外数据集上分别以+20%和+7%的F1分数稳定超越基线方法，证明OAT是诊断代理系统故障的一种有前景且高效的方向。

English

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 times faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.