AgentLens: 面向编码智能体评估的生产环境评估轨迹审查

摘要

我们推出AgentLens——一个面向生产环境评估的交互式代码智能体基准测试。大多数代码智能体基准测试将一次运行简化为单一指标——任务是否通过？——但实际上，使用这些智能体的用户会经历完整轨迹：智能体如何遵循指令、使用工具、验证自身工作、从错误中恢复，以及在此过程中与用户的交互方式。AgentLens评估的就是这个完整轨迹。它结合了形式化验证（存在客观校验）与LLM撰写的轨迹审查及并排对比，使每次运行都能生成对评分依据的可读性解释。这使得AgentLens的用途不仅限于模型排名：我们用它来诊断模型行为、对比自家智能体的连续版本，并通过夜间评估管线捕捉产品性能退化。我们已将该基准测试开源，地址为https://github.com/agent-lens/agent-lens-bench。

English

We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.