AgentCompass：智能体能力的统一评测基础设施

AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

July 15, 2026

作者: Zichen Ding, Jiaye Ge, Shufan Jiang, Kai Chen, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu

cs.AI

摘要

随着大型语言模型（LLMs）向自主智能体演进，构建统一的评估基础设施变得至关重要。然而，当前的评估流程高度碎片化且紧密耦合，既阻碍了可重复性，又造成了工程冗余。为解决这一问题，我们提出了AgentCompass——一款开源、轻量级且可扩展的LLM智能体评估基础设施。AgentCompass将评估流程分解为三个独立组件，即基准测试（Benchmark）、测试框架（Harness）和环境（Environment），从而在无需重新实现复杂执行逻辑的前提下实现灵活配置。此外，它采用具备容错机制的异步运行时系统，并配备全面的轨迹分析工具，可透明地诊断诸如奖励漏洞（reward-hacking）等细微故障模式。AgentCompass原生支持覆盖五大能力维度的20余项基准测试，为社区提供了可扩展且可重复的基础设施，以推动智能体研究的发展。

English

As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.