一次成功不等于可靠：Thinkingbox——面向有状态业务流程智能体的沙盒与基准

摘要

最近的智能体基准测试越来越倾向于将评估嵌入可执行环境中，涵盖从代码修复到网页导航、应用API和函数调用等场景。然而，完成代码之外的重要工作需要的不仅是生成看似合理的响应或有效的工具调用：智能体必须通过多轮交互收集缺失信息，遵循领域策略，协调相互依赖的工具，并在不产生附带影响的情况下实现正确的持久状态转换。在本文中，我们介绍了Thinkingbox，一个用于工具-智能体-用户交互的沙箱环境，提供隔离的MCP兼容工具会话、完整的执行轨迹以及基于终端后端状态的结果评估。基于该沙箱，Thinkingbox-bench包含了507个策略条件化工作流，覆盖零售、酒店、汽车保险、新银行内部IT以及咨询IT/人力资源支持等多种场景。每次尝试都通过任务特定的可执行检查进行评估，这些检查接受有效轨迹，同时拒绝错误、缺失或多余的效果；指定任务还会额外检查最终响应的必要属性。在专有模型和开放权重模型中，最强模型达到了65.36%的pass@1，但pass^20仅为25.25%。此外，许多失败的试验表现出干净的终止和有效的状态改变动作，这表明响应或工具调用级别的信号并不能清晰反映端到端任务的完成情况。Thinkingbox-bench揭示了一个巨大差距：偶然找到成功轨迹与可靠完成有状态业务任务之间的差距。我们已发布Thinkingbox和Thinkingbox-Bench：https://github.com/microsoft/thinkingbox

English

Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox