AgentLens：基於生產評估的軌跡審查用於編碼代理評估

摘要

我們提出 AgentLens，一個基於生產評估的互動式程式碼智慧體基準測試。多數程式碼智慧體基準測試將一次執行簡化為單一位元——任務是否通過？——但實際使用這些智慧體的人們體驗的是整個軌跡：智慧體如何遵循指令、使用工具、驗證自身工作、從錯誤中恢復，以及沿途與使用者交流。AgentLens 評估整個軌跡。它將存在客觀檢查的形式化驗證，與 LLM 撰寫的軌跡審查及並排比較配對，使每次執行都能產出一份可讀的解釋，說明為何得到該評分。這讓 AgentLens 不僅用於模型排名：我們用它來診斷模型行為、比較我們自身智慧體的連續版本，並在夜間評估管線中捕捉產品回歸。我們在 https://github.com/agent-lens/agent-lens-bench 以開源形式發布該基準測試。

English

We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.