持续图形用户界面代理

摘要

随着数字环境（数据分布）处于动态变化中，新GUI数据随时间不断涌现——引入新的领域或分辨率——在静态环境中训练的智能体性能会出现退化。本研究提出"持续GUI智能体"新任务，要求GUI智能体在领域和分辨率变化的条件下实现持续学习。我们发现，由于动态场景中用户界面交互点和交互区域的多样性，现有方法在GUI分布随时间变化时难以保持稳定的定位基准。为此，我们提出动态锚定GUI框架（GUI-AiF），这是一种通过强化微调实现持续学习稳定的新框架，其核心是两种新型奖励机制：动态锚点奖励（APR-iF）与动态锚域奖励（ARR-iF）。这些奖励机制引导智能体与动态变化的交互点和区域保持对齐，有效缓解现有奖励策略对静态定位线索（如固定坐标或元素尺寸）的过度适应问题。大量实验表明GUI-AiF超越了现有最优基线方法。本研究首次建立了GUI智能体的持续学习框架，揭示了强化微调在持续GUI智能体领域尚未开发的潜力。

English

As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.