持续学习的图形用户界面智能体

摘要

随着数字环境（数据分布）的动态变化，新图形用户界面数据持续涌入——引入新领域或分辨率——在静态环境中训练的智能体性能会逐渐退化。本研究提出"持续GUI智能体"新任务，要求GUI智能体在领域和分辨率变迁下实现持续学习。我们发现，由于动态场景中用户界面交互点和交互区域的多样性，现有方法难以在GUI分布变化时保持稳定的定位基准。为此，我们提出动态锚定GUI框架（GUI-AiF），这是一种通过强化微调实现持续学习稳定的新框架，其核心是两种新型奖励机制：动态锚点奖励（APR-iF）与动态锚域奖励（ARR-iF）。这些奖励机制引导智能体与动态变化的交互点及区域保持对齐，有效克服现有奖励策略过度依赖静态定位基准（如固定坐标或元素尺寸）的缺陷。大量实验表明GUI-AiF超越了现有最优基线方法。本研究开创了首个面向GUI智能体的持续学习框架，揭示了强化微调技术在持续GUI智能体领域尚未开发的潜力。

English

As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.