MemHarness：记忆是被重建的，而非回放的

摘要

检索过往经验已成为增强大语言模型智能体的常用策略。然而，现有的大多数记忆增强智能体将检索到的经验视为要逐字重放的静态记录，无论其是否与智能体当前情境相符，都直接注入上下文。这种“重放”范式忽视了存储经验的抽象性与普适性同决策时所面临的具体且不断变化的状态之间的差距，常常导致负迁移。相比之下，人类很少逐字回忆过往经验；相反，他们会根据当前情境重组和调整检索到的记忆。受此启发，我们提出MemHarness框架，使大语言模型智能体能够基于当前情境主动利用并重构过往经验。在每个决策步骤中，统一策略模型会根据当前状态对检索到的经验进行评判与重构，在行动之前生成基于上下文的指导。这种重构能力通过GRPO的端到端训练自然涌现。在ALFWorld和WebShop上的实验表明，MemHarness显著优于纯强化学习基线和静态记忆增强基线，在分布外（OOD）场景中展现出较强的鲁棒性。此外，我们的分析表明，这种重构目标不仅防止了负迁移，还在训练期间起到潜在引导的作用，从根本上提升了智能体自身的推理能力。

English

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.