论计算机使用代理的可靠性

摘要

计算机使用智能体在网页导航、桌面自动化及软件交互等现实任务中快速提升，某些场景下甚至超越人类表现。然而即使任务与模型保持不变，一次成功的智能体在重复执行相同任务时仍可能失败。这引出一个根本性问题：若智能体能够成功执行某任务一次，为何无法持续稳定地复制该表现？本研究通过三个维度探究计算机使用智能体不可靠性的根源：执行过程中的随机性、任务描述的模糊性以及智能体行为的多变性。我们在OSWorld环境中通过重复执行相同任务，结合能捕捉跨设置任务级变化的配对统计检验进行分析。研究表明，可靠性既取决于任务描述方式，也受智能体跨次执行行为变化的影响。这些发现启示我们：需在重复执行中评估智能体性能，允许智能体通过交互消除任务模糊性，并优先选择跨次运行保持稳定的策略。

English

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.

论计算机使用代理的可靠性

On the Reliability of Computer Use Agents

摘要

Support