一次成功不代表可靠性：Thinkingbox——針對有狀態商業工作流程中智慧體之沙盒與基準測試

摘要

近期的代理基準（agent benchmarks）日益將評估立基於可執行的環境中，涵蓋範圍從程式碼修復到網頁導航、應用程式介面（API）與函式呼叫。然而，完成程式碼以外的具影響性工作，遠不止於產出看似合理的回應或有效的工具呼叫：代理必須在多輪互動中收集遺漏的資訊、遵循領域政策、協調相互依賴的工具，並在不產生附帶影響的情況下實現正確的持久狀態轉換。在本文中，我們介紹 Thinkingbox，一個用於工具—代理—使用者互動的沙盒環境，提供隔離的 MCP 相容工具工作階段、完整的執行軌跡，以及對終端後端狀態的結果評估。基於此沙盒，Thinkingbox-bench 包含橫跨多種情境的 507 個政策條件化工作流程，涵蓋零售、旅宿業、汽車保險、純網銀內部資訊科技，以及顧問公司的資訊科技／人力資源支援等領域。每次嘗試皆由任務特定的可執行檢查進行評估，這些檢查接受有效的軌跡，同時拒絕錯誤、遺漏或額外的影響；指定的任務還會額外檢查最終回應的必需屬性。在專有與開放權重模型中，最強的模型達到 65.36% 的 pass@1，但僅有 25.25% 的 pass^20。此外，許多失敗的試驗顯示正常終止與有效的狀態變更動作，這表明回應或工具呼叫層級的信號並不能清晰代理端對端任務的完成情況。Thinkingbox-bench 揭示了偶爾找到成功軌跡與可靠完成具狀態商業任務之間的巨大差距。我們發布 Thinkingbox 與 Thinkingbox-Bench：https://github.com/microsoft/thinkingbox

English

Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox