UI-Mate：通过上下文演示推进开放权重基础GUI智能体

摘要

基础GUI智能体可以自动化处理复杂的数字任务，但其部署受到训练数据稀缺且有偏、提示模糊以及执行不可靠等因素的阻碍。日常业务流程依赖于用户特定的工具和隐性约定，因此未明示的指令可能导致不同运行之间产生任意差异。我们提出UI-Mate，一个将基于环境的训练栈与上下文演示学习相结合的基础GUI智能体。UI-Mate做出三项贡献：可扩展的基于环境的训练栈：一个闭环数据引擎通过统一的任务-验证器捆绑包，在超大规模并行环境中自动化任务生成、环境构建、展开（rollout）、过滤、能力平衡、SFT和在线RL。上下文演示学习：一种将多模态演示转化为灵活的子任务级工作流、遵循相关演示步骤并从实时界面重新规划的机制。OSWorkerBench基准与洞察：一个包含41个应用中100项长周期办公任务的基准，支持纯指令和演示引导两种评测方式。其演示资源将33项任务的自演示设置与45项任务的变体演示设置区分开：前者由同一目标的成功强智能体展开轨迹构建，后者由相关但不相同任务的人类操作录制构建。实验表明，UI-Mate-27B在通用计算机使用基准上取得了开源权重模型的最新最优结果，在OSWorld-Verified上得分为77.0%，在WindowsAgentArena上得分为66.2%。在OSWorkerBench上，它达到41.0%的严格成功率和76.9%的进度，分别比其Qwen3.6-27B基础模型高出17.7和24.5个百分点。在33项任务的自演示子集上，单条演示将严格成功率从17.2%提升至35.4%，进度从67.9%提升至81.1%，大幅提高了长周期任务的可靠性。项目页面：https://ui-mate.github.io。

English

Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.