ClawProBench：結合運行時覆蓋率與凍結職場式保留集之軌跡感知AI代理評測

摘要

代理（Agent）基準測試通常僅評估最終答案，即使代理是在有狀態的運行時（stateful runtime）上執行。我們主張這對被評估的對象定義不足：適當的評估單位應是「已宣告的模型加運行時配置」（declared model-plus-runtime configuration），其失敗可能發生於證據獲取、運行時路由、安全邊界或重複執行等環節。我們提出 ClawProBench，一個軌跡感知（trace-aware）的基準測試，用於運行時原生的代理評估，並實例化於 OpenClaw——一個具備工作區工具及瀏覽、記憶、訊息、排程、技能與子代理等原生介面的即時代理運行時。ClawProBench 定義兩條評測軌道：其一為包含 102 個情境的完整輪廓（full profile），涵蓋即時工作區與原生運行時路由任務；其二為包含 68 個情境的凍結保留集（holdout），採用封閉世界 JSON 輸出契約以進行穩健排序。試驗透過安全閘控（safety-gated）公式，從執行軌跡中結合正確性、過程品質與效率進行評分，並保留失敗證據以供稽核。我們提供的匿名工件包含基準測試定義、評分程式碼、清單與去識別化軌跡。我們在完整輪廓上評估 68 種配置，在保留集上評估 37 種配置。最高安全閘控平均軌跡分數為 0.7671。原生運行時任務的表現低於工作區即時任務（0.5238 對 0.6415）。在保留集上，pass@k-any 優於嚴格三次試驗通過率（0.6638 對 0.2890），而完整輪廓與保留集的排名顯示弱對齊（斯皮爾曼相關係數 0.1300）。僅基於正確性的排名與過程感知、安全閘控及嚴格通過的觀點存在顯著差異。僅以最終答案為依據的排行榜可能隱藏原生介面的弱點、一次性成功以及軌跡局部的代理失敗模式。

English

Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.