StateAct：面向长時程電腦使用代理的程式狀態優先於像素

摘要

使用電腦的智能體通常透過強化感知能力來改進：更好的模型用於讀取螢幕截圖並選擇點擊位置。然而，螢幕截圖僅是底層程式狀態（例如，包含任務資料的檔案、應用後端和DOM）的一種有損渲染。不同的狀態可能產生相同的像素，而程式碼可以直接檢查和修改該狀態。StateAct 是基於這種區別而構建的代碼優先、多智能體框架。其主要智能體通過使用程式碼直接與程式狀態交互，而專用的GUI子智能體則在少數需要它的子目標上處理螢幕截圖與點擊互動——僅佔108項任務中的28項，以及主要智能體步驟的1.1%。對程式狀態的直接存取也支援驗證：一個獨立的完成閘門會雙重檢查保存結果是否存在結構性錯誤，例如輸出遺失、未保存或寫入錯誤路徑。為了在數百個步驟中保持正確方向，主要智能體將子目標交給新的子智能體處理，保持自身上下文的專注。在 OSWorld 2.0 上，StateAct 將 Claude Opus 4.8 的二元成功率從20.6%提升至26.9%，部分成功率從54.8%提升至61.6%，且每項任務的成本約為僅使用螢幕截圖驅動相同模型的九分之一；沒有GUI子智能體的純代碼變體僅達到45.9%的部分成功率，低於基於螢幕截圖的基準線54.8%。總體而言，將行動、驗證和記憶接地於狀態（我們稱之為狀態接地）將主要瓶頸從感知轉向推理：失敗更多取決於智能體的想法而非所見。

English

Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.