StateAct: 程序状态，先于像素，用于长时程计算机使用智能体

摘要

计算机使用智能体通常通过强化感知能力来改进：构建更好的模型来读取截图并选择点击位置。然而，截图仅仅是底层程序状态（例如持有任务数据的文件、应用后端和DOM）的有损渲染。不同的状态可能生成相同的像素，而代码可以直接检查并修改该状态。StateAct正是基于这一区别构建的、以代码优先的多智能体框架。其主智能体通过使用代码直接与程序状态交互，而专门的GUI子智能体仅在少数需要截图与点击交互的子目标中发挥作用——在108个任务中仅有28个需要，且占主智能体步骤的1.1%。对程序状态的直接访问同样支持验证：独立的完成门控会双重检查保存结果是否存在结构性失败，例如输出缺失、未保存或写入错误路径。为了在数百个步骤中保持聚焦，主智能体将子目标交由新的子智能体处理，从而保持自身上下文的专注。在OSWorld 2.0上，StateAct将Claude Opus 4.8的二值成功率从20.6%提升至26.9%，部分成功率从54.8%提升至61.6%，而每个任务的成本比单纯依赖截图的同一模型低约9倍；无GUI子智能体的纯代码变体仅达到45.9%的部分成功率，低于基于截图的基线（54.8%）。总体而言，将行动、验证和记忆基于状态（我们称之为状态基础化）将主要瓶颈从感知转向推理：失败更多地取决于智能体的思考而非其视觉感知。

English

Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.