從成功之流中追溯能動性失敗

摘要

針對基於LLM的代理系統進行故障歸因，亦即識別失敗軌跡中導致任務失敗的具體步驟，對於除錯與系統改進至關重要。現有方法若非仰賴基於提示（prompting）的流程（此類方法運算成本高昂），便是需要在包含步驟層級錯誤標註的失敗軌跡上進行後訓練（post-training），而此類標註資料不僅收集成本高昂，更難以擴展。我們主張，實用的故障歸因模型應具備輕量化特性，且能在無步驟層級監督下僅以失敗資料進行訓練。為此，我們聚焦於無監督故障歸因問題：僅以成功軌跡進行訓練，並在推理階段針對給定的失敗軌跡識別錯誤步驟。我們提出OAT方法，將此問題轉化為基於神經控制微分方程式（neural controlled differential equations）的單類學習（one-class learning），在潛在空間中建模成功軌跡的動態模式。在推論階段，根據每個步驟偏離成功軌跡學習所得動態的程度，賦予異常分數（anomaly score），進而建構錯誤步驟集合。實驗結果顯示，僅需訓練100條成功軌跡，OAT的執行速度即比基於提示的基準方法快200至5000倍，同時在領域內（in-domain）與分布外（out-of-distribution）資料集上，F1分數分別穩定高出20%與7%，證實OAT為診斷代理系統故障提供了一條具潛力且高效的發展方向。

English

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 times faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.