GUICrafter: 利用大量未标注屏幕截图的弱监督GUI智能体

摘要

数据作为现代智能的基础基质，极大推动了当前基础模型的发展。研究者们自然希望将这一范式拓展至图形用户界面（GUI）代理领域，试图通过类似范式构建强大的GUI代理。然而，GUI代理数据无法直接从互联网获取，导致其规模化采集成本高昂且困难重重。因此，当前GUI代理面临跨设备泛化能力弱、细粒度界面元素视觉定位能力不足等困境。为应对GUI代理中的数据挑战，我们提出GUICrafter——一种利用海量无标注截图大幅降低对昂贵人工标注依赖的弱监督GUI代理。GUICrafter探索了通过两个递进阶段训练GUI代理的课程学习框架：首先，模型从大规模无标注截图和网页中学习视觉定位，利用GUI交互中蕴含的丰富上下文信号实现无需人工标注的学习；随后在第二阶段，我们借助少量高质量数据通过强化学习对模型进行校准。实验表明，GUICrafter在仅使用UI-TARS系统0.1%数据量的情况下，即可达到甚至超越该先进系统的性能。此外，在相同标注数据量下，GUICrafter全面超越GUI-R1等此前所有方法。相关代码、数据及模型已开源至https://github.com/fansunqi/GUICrafter。

English

Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.