GUICrafter：利用大量未標注截圖的弱監督圖形使用者介面代理

摘要

數據作為現代智能的基本載體，極大推動了當前基礎模型的發展。研究者自然希望將此範式延伸至GUI代理領域，期待透過類似範式建構強大的GUI代理。然而，GUI代理的數據無法直接從網際網路擷取，導致其大規模收集成本高昂且困難重重。因此，現有GUI代理普遍存在跨裝置泛化能力薄弱、對細粒度GUI元素的視覺定位能力有限等問題。為解決GUI代理的數據挑戰，我們提出GUICrafter——一種利用大量未標註螢幕截圖的弱監督GUI代理，大幅降低對昂貴人工標註的依賴。GUICrafter探索基於課程學習的框架，透過兩個漸進階段訓練GUI代理：首先，模型從大規模未標註螢幕截圖與網頁中學習視覺定位，藉助GUI互動中蘊含的豐富上下文線索，無需人工標註；接著在第二階段，我們運用少量高品質數據，透過強化學習對模型進行校準。實驗結果顯示，GUICrafter在僅使用UI-TARS 0.1%數據量的情況下，即可達到甚至超越其效能。此外，在相同標註數據量條件下，GUICrafter全面優於GUI-R1等先前方法。程式碼、數據與模型已公開於 https://github.com/fansunqi/GUICrafter。

English

Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.