关于计算机使用代理的可靠性探讨

摘要

計算機操作智能體在網頁瀏覽、桌面自動化和軟件交互等實際任務中快速提升，某些情況下甚至超越人類表現。然而即使任務與模型保持不變，成功執行一次的智能體在重複執行相同任務時仍可能失敗。這引發了一個根本性問題：若智能體能成功完成某項任務，究竟是什麼阻礙了其穩定發揮？本研究通過三個維度探討計算機操作智能體的不可靠性來源：執行過程中的隨機性、任務描述的模糊性，以及智能體行為的波動性。我們在OSWorld環境中對同一任務進行重複執行實驗，並採用配對統計檢驗來捕捉不同設置下的任務級變化。分析表明，可靠性既取決於任務描述方式，也受智能體跨次執行行為變化的影響。這些發現提示我們需要通過重複執行來評估智能體性能，允許智能體通過交互消除任務模糊性，並優先選擇跨次運行保持穩定的策略。

English

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.

关于计算机使用代理的可靠性探讨

On the Reliability of Computer Use Agents

摘要

Support