經受考驗：重新評估智能體在陌生環境中的能力

摘要

随着代理系统的持续演进并在实际场景中得到广泛应用，对其能力进行忠实评估的需求日益增长。然而，当前的基准测试通常构建于热门应用之上，任务相对简单，仅聚焦于少数能力维度，而忽视了更广泛的范畴，导致现代代理系统的性能趋于饱和，难以探知其局限性。为此，我们推出GauntletBench——一个基于网页的基准测试，旨在评估代理系统在挑战性场景中的泛化能力，重点关注三个尚未充分探索的能力维度（时间感知、图形理解与3D推理），覆盖五个较少涉及的职业应用领域（视频编辑器、工作流构建器、3D建模师、航班分析员与电路设计师），每个领域包含20项视觉密集型任务（共计100项）。我们的基准测试采用模块化流水线设计：包含兼容开源与闭源代理框架的环境、受控的网页应用、结构化的任务套件，以及配备多种指标的自动化评估引擎。与普遍预期相反，实证结果表明，前沿代理系统远未达到人类水平。即使是最先进的代理系统，在GauntletBench上也仅取得19.1%的成功率，揭示了这些被忽视能力维度及其泛化性的局限。相比之下，非专业人类标注者在我们设计得具有挑战性但切实可行的任务上取得了超过80%的成功率，这凸显了当前代理能力与复杂现实场景要求之间的巨大鸿沟。

English

As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchmarks are typically built on popular applications with relatively simple tasks and focus on a narrow set of capabilities while overlooking broader dimensions, resulting in saturated performance on modern agents and failing to probe their limitations. To this end, we introduce GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), across five less-covered professional applications (Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer), each with 20 vision-intensive tasks (100 in total). Our benchmark provides a modular pipeline that comprises an environment compatible with both open- and closed-source agent frameworks, a controlled web-based application, a well-structured task suite, and an automated evaluation engine with diverse metrics. Contrary to widespread expectations, our empirical results reveal that frontier agentic systems remain far from achieving human-level performance. Even the state-of-the-art agent achieves only a 19.1% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation. By comparison, non-expert human annotators achieve over 80% success on our challenging yet feasible tasks, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.