FocusMem：分解潛在GUI記憶中的內容、讀出與信任

摘要

GUI 智能體必須同時記住先前任務中習得的有效經驗，以及當前互動中尚未完成的進度。潛在記憶透過將多模態軌跡壓縮為少量連續向量，提供了一種緊湊的解決方案。然而，現有方法通常將每個軌跡對應至一個固定的記憶區塊，並主要透過下一步動作監督來訓練。這造成了三個實際問題：壓縮過程中可能遺失重要細節、同一個記憶區塊必須服務於不同的決策階段，以及不相關的檢索軌跡仍可能誤導智能體。我們提出 FocusMem，在一個緊湊的潛在記憶介面中分離這些職責。角色感知的內容基底促使情節記憶保留可重用的經驗，而工作記憶則保留任務進度。狀態條件化讀取對相同的儲存證據產生決策特定的視圖，同時一個輕量級的信任閘門可抑制與當前步驟看似不相關的記憶區塊。所有元件均在 GUI 策略凍結的情況下進行訓練。在五個 GUI 智能體基準測試中，FocusMem 持續優於完全匹配的僅動作固定記憶基線，以及先前的潛在記憶改編方法。進一步分析顯示，語意監督與功能監督保留了互補的資訊；狀態條件化讀取在周圍軌跡上下文增長時表現得更為穩健；信任閘門則減少了注入不相關情節證據所造成的危害。這些結果表明，有效的潛在記憶不僅取決於對過去互動的壓縮，還取決於保留了什麼、暴露了什麼、以及允許了什麼。

English

GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map each trajectory to one fixed memory block and train it mainly through next-action supervision. This creates three practical problems: important details may be lost during compression, the same memory block must serve different decision stages, and irrelevant retrieved trajectories may still mislead the agent. We introduce FocusMem, which separates these responsibilities within a compact latent-memory interface. A role-aware content basis encourages episodic memory to retain reusable experience and working memory to retain task progress. A state-conditioned readout generates a decision-specific view of the same stored evidence, while a lightweight trust gate can suppress memory blocks that appear irrelevant to the current step. All components are trained while the GUI policy remains frozen. Across five GUI-agent benchmarks, FocusMem consistently outperforms a fully matched action-only fixed-memory baseline and prior latent memory adaptations. Further analysis shows that semantic and functional supervision preserve complementary information, state-conditioned readout is more robust as surrounding trajectory context grows, and the trust gate reduces the harm caused by injected irrelevant episodic evidence. These results show that effective latent memory depends not only on compressing past interaction, but also on what is retained, what is exposed, and what is allowed.