FocusMem：潜在GUI记忆中的内容、读出与信任因子化

摘要

GUI智能体必须同时记住先前任务中积累的有用经验以及当前交互中未完成的进度。潜在记忆通过将多模态轨迹压缩为少量连续令牌，提供了一种紧凑的解决方案。然而，现有方法通常将每条轨迹映射到一个固定的记忆块，并主要通过下一动作监督进行训练。这产生了三个实际问题：压缩过程中重要细节可能丢失、同一记忆块必须服务于不同的决策阶段、以及检索到的无关轨迹仍可能误导智能体。我们提出FocusMem，在紧凑的潜在记忆接口内分离这些职责。角色感知的内容基础鼓励情景记忆保留可复用的经验，工作记忆保留任务进度。状态条件化读出机制对同一存储证据生成决策特定的视图，而轻量级信任门控可抑制与当前步骤似乎无关的记忆块。所有组件均在GUI策略保持冻结的情况下进行训练。在五个GUI智能体基准测试中，FocusMem持续优于完全匹配的仅动作固定记忆基线和先前的潜在记忆改编方法。进一步分析表明，语义与功能监督保留了互补信息，状态条件化读出在周围轨迹上下文增长时更具鲁棒性，信任门控减少了注入无关情景证据造成的危害。这些结果表明，有效的潜在记忆不仅依赖于压缩过去的交互，还取决于保留什么、暴露什么以及允许什么。

English

GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map each trajectory to one fixed memory block and train it mainly through next-action supervision. This creates three practical problems: important details may be lost during compression, the same memory block must serve different decision stages, and irrelevant retrieved trajectories may still mislead the agent. We introduce FocusMem, which separates these responsibilities within a compact latent-memory interface. A role-aware content basis encourages episodic memory to retain reusable experience and working memory to retain task progress. A state-conditioned readout generates a decision-specific view of the same stored evidence, while a lightweight trust gate can suppress memory blocks that appear irrelevant to the current step. All components are trained while the GUI policy remains frozen. Across five GUI-agent benchmarks, FocusMem consistently outperforms a fully matched action-only fixed-memory baseline and prior latent memory adaptations. Further analysis shows that semantic and functional supervision preserve complementary information, state-conditioned readout is more robust as surrounding trajectory context grows, and the trust gate reduces the harm caused by injected irrelevant episodic evidence. These results show that effective latent memory depends not only on compressing past interaction, but also on what is retained, what is exposed, and what is allowed.