UniClawBench：面向真實世界任務的主動式代理通用基準

摘要

大型語言模型與多模態大型語言模型的快速發展，加速了具備主動能力的智能體（Agent）的出現，這些智能體能夠操作日常工具並在真實環境中協助用戶。然而，現有基準測試難以有效評估此類智能體，原因在於它們通常依賴沙盒化環境與單輪評估模式。此外，其基於場景的任務分類將多種模型能力混雜在同一任務類別中，使得智能體失敗的根本原因難以辨識。為了解決這些限制，我們提出UniClawBench，這是首個以能力為導向的基準測試，旨在動態真實環境中評估主動智能體。UniClawBench圍繞五項基礎模型能力建構：技能運用、探索、長文本推理、多模態理解與跨平台協調。基於這些能力，我們設計了400項雙語真實世界任務。不同於依賴靜態預錄答案的既有基準測試，我們的基準測試在即時Docker容器中，透過細粒度、逐步完成的檢查點來評估智能體。此外，我們設計了一套閉環評估策略，包含執行智能體、隱藏監督智能體與用戶智能體，以模擬真實多輪人類反饋，同時不洩漏評分標準。為了將基礎模型能力與框架層級的設計選擇區分開來，我們在多個智能體框架下評估了當前最先進的模型。透過模型與框架間的全面比較，我們展示基礎模型能力與智能體框架設計如何共同影響真實環境中的表現。為促進未來研究，我們將基準測試與程式碼公開於 https://github.com/HKU-MMLab/UniClawBench。

English

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.