Agent S：一個開放的代理框架，讓電腦像人類一樣運作。

Agent S: An Open Agentic Framework that Uses Computers Like a Human

October 10, 2024

作者: Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang

cs.AI

摘要

我們提出了Agent S，一個開放的代理框架，通過圖形用戶界面（GUI）實現與計算機的自主交互，旨在通過自動執行複雜的多步任務來改變人機交互。Agent S的目標是應對自動執行計算機任務中的三個關鍵挑戰：獲取特定領域知識、規劃長期任務視野以及處理動態、非統一界面。為此，Agent S引入了經驗增強的階層規劃，通過在多個層次上從外部知識搜索和內部經驗檢索中學習，促進有效的任務規劃和子任務執行。此外，它利用一個代理-計算機界面（ACI）來更好地引出基於多模式大型語言模型（MLLMs）的GUI代理的推理和控制能力。在OSWorld基準測試中的評估顯示，Agent S在成功率上優於基準線9.37%（相對改進了83.6%），並實現了新的最先進技術。全面分析突出了各個組件的有效性，並為未來改進提供了見解。此外，Agent S在新發布的WindowsAgentArena基準測試中展示了對不同操作系統的廣泛泛化能力。代碼可在https://github.com/simular-ai/Agent-S找到。

English

We present Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.