MemHarness：記憶是被重建的，而非重播的

摘要

檢索過往經驗已成為增強大型語言模型代理的常用策略。然而，現有的記憶增強代理大多將檢索到的經驗視為靜態紀錄，以逐字重播的方式注入情境中，而不論這些經驗是否與代理當前面臨的情境相符。這種「重播」模式忽略了儲存經驗的抽象概括本質與決策當下具體多變狀態之間的鴻溝，常導致負向遷移。相比之下，人類鮮少逐字回顧過往經驗，而是會重新組織並調整檢索到的記憶，以適應當前情境。受此啟發，我們提出 MemHarness——一個使大型語言模型代理能根據當下情境主動利用並重建過往經驗的框架。在每個決策步驟中，統一的策略模型會依據當前狀態對檢索到的經驗進行批判性評估與重建，在行動前產生植根於情境的引導。此重建能力可透過 GRPO 的端對端訓練自然湧現。在 ALFWorld 與 WebShop 上的實驗顯示，MemHarness 顯著優於純強化學習與靜態記憶增強的基準方法，在分布外（OOD）情境中展現出強大的穩健性。此外，我們的分析揭示，此重建目標不僅能防止負向遷移，在訓練期間更能作為潛在引導，從根本層面提升代理的內在推理能力。

English

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.