τ^τ-Bench：一種用於端到端、真實智能體構建的環境

摘要

LLM 代理正迅速成為生產級軟體，被部署用於處理客戶服務、裁決爭議以及營運內部系統。值得注意的是，構建這些代理的工作日益交由編碼代理處理，然而現有基準測試對於一個 AI 系統能否在真實客戶委託的條件下交付此類代理，所能說明的相當有限。我們提出 τ^τ-bench（讀作 hyper-tau-bench），這是一個以代理構建為任務的基準測試。開發代理會獲得企業實際留存的記錄、持有需求的客戶、營運必須經由的生產環境 API、需繼承的程式碼庫，以及服務成本與模型使用上的限制——這正是真實委託案所給予的起點。代理必須據此交付一個完整的客戶服務代理，其評分方式是將該代理部署於保留的模擬用戶進行測試。在橫跨四個領域的 53 項任務中，最強的配置——Claude Code 中的 Claude Opus 5——僅通過 23.9% 的評估模擬。與此同時，專家撰寫的參考基準上限得分為 82.2%。這些失敗與人類代理開發者所見的情況如出一轍：模型以淺層查詢取代對記錄的深度理解，幾乎不與客戶溝通，對於代理架構與服務支出的實驗也過少，往往直接交付第一個能運行的設計即止步。我們期望 τ^τ-bench 能將協作式代理構建的工作轉化為編碼代理可量測的目標。

English

LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce τ^τ-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for τ^τ-bench to turn the work of cooperative agent building into a measurable target for coding agents.