A²E：端到端智能體審計引擎

摘要

隨著大型語言模型（LLMs）的快速發展，框架（harnesses）已成為在各個領域部署智能體（agents）的關鍵基礎設施。快速演進的框架生態系統也使得嚴謹的能力評估變得日益重要。然而，高效地建構一個端對端、系統化且全面的評估流程仍然是一項重大挑戰。為應對此挑戰，我們提出了 A^2E（Agent Auditing Engine），一個專為智能體框架設計的端對端評估引擎。A^2E 利用我們新提出的智能體任務協議（Agent Task Protocol, ATP），實現不同框架間評估任務的快速整合。透過自動儀器化的監控器（Monitor），A^2E 在實驗過程中擷取並生成標準化執行軌跡。在評估階段，A^2E 使用一套多維度指標系統性地評估框架能力。與僅考量正確性相比，這些指標在執行效率、工具使用、任務規劃和錯誤恢復方面提供了更細粒度的框架間差異特徵。使用 A^2E 進行的實驗進一步揭示，模型-框架組合在不同類型的任務上表現出顯著的性能差異，且沒有任何單一組合能在所有任務中持續優於其他組合。這些發現不僅證明了系統性評估的必要性，也為模型與框架的協同演進提供了有益的指導。我們的程式碼可在 https://github.com/datamllab/A2E 取得。

English

With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. A^2E leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.