RoMeRL：透過降階效用狀態平衡自演化智能體記憶中的回饋覆蓋與記憶-獎勵陷阱

摘要

針對自我進化的 LLM 代理所設計的基於學習的記憶系統，面臨兩個密切相關的挑戰。首先，軌跡索引的效用隨互動歷史增長，從而將有限的回饋分散到持續擴張的狀態空間中。其次，由於軌跡層級的獎勵被共同分配給共同檢索到的記憶，不相關的經驗可能接收到誤導性的效用更新，因而陷入記憶獎勵陷阱。為解決這些挑戰，我們提出了降階記憶強化學習（RoMeRL），它使用按結果極性與記憶動態分解的固定維度每任務記憶狀態，來表示持續增長的軌跡索引效用空間。RoMeRL 透過一組固定的語義座標整合新經驗，這些座標的內容會隨時間更新或替換，從而將回饋集中於有界的效用支撐上。在理論上，我們證明這種降階參數化能提高每個效用座標所收到的平均回饋，並在通用座標轉換模型下刻畫了錯誤座標的穩態佔有率。在實證上，於 ALFWorld 與 LifelongAgentBench 中，RoMeRL 提升了任務效能，將 Cold-Q 比率降低了 80.0%，將回饋密度提高了約 6.0 倍，將所維護的記憶大小減少了 84.4%，並減少了 21.1% 的 LLM 呼叫。這些結果表明，降階效用狀態在限制持續性獎勵污染的同時，支援了高效的自我進化代理記憶。程式碼可於以下網址取得：https://github.com/YOUNG-fnxm/RoMeRL

English

Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL