ClawProBench：基于轨迹感知的AI智能体评估——结合运行时覆盖率与冻结工作场景式保留集

摘要

智能体基准测试常常只评估最终答案，即便智能体运行在有状态运行时之上。我们认为这欠定义了被评估的对象：恰当的评估单元应是声明的模型加运行时配置，其故障可能发生在证据获取、运行时路由、安全边界或重复执行之中。我们提出 ClawProBench，一个面向运行时原生智能体评估的轨迹感知基准测试，实例化于 OpenClaw 之上——一个具备工作区工具以及浏览、记忆、消息传递、调度、技能和子智能体等原生接口的实时智能体运行时。ClawProBench 定义了两个轨道：一个包含 102 个场景的完整配置档，涵盖实时工作区与原生运行时路由任务；以及一个冻结的 68 场景留出集，采用封闭世界的 JSON 输出契约以实现稳健排名。试验通过安全门控公式从执行轨迹中评分，该公式综合了正确性、过程质量和效率，并为审计保留故障证据。我们的匿名工件包含基准定义、评分代码、清单文件和经脱敏处理的轨迹。我们在完整配置档上评估了 68 种配置，在留出集上评估了 37 种。最高的安全门控平均轨迹得分为 0.7671。原生运行时任务的表现低于实时工作区任务（0.5238 对 0.6415）。在留出集上，pass@k-any 优于严格的三轮通过（0.6638 对 0.2890），而完整配置档与留出集之间的排名一致性较弱（斯皮尔曼相关系数 0.1300）。纯基于正确性的排名与过程感知、安全门控及严格通过视角下的排名存在显著差异。仅关注最终答案的排行榜可能掩盖原生接口的弱点、一次性成功以及轨迹局部的智能体故障模式。

English

Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.