机器人操作中视觉-语言-动作模型的双重潜在记忆

摘要

主流视觉-语言-动作（VLA）模型在马尔可夫假设下主要依据当前观测预测动作，从而难以处理依赖时间的长时域任务。现有记忆增强型VLA模型或扩展观测窗口，或从记忆库中检索历史信息作为辅助策略端上下文，但这些方法将记忆置于VLA推理的原生潜在嵌入空间之外，导致历史经验无法与多模态推理及动作生成流畅交互。为此，我们提出LaMem-VLA——一种原生嵌入潜在记忆的框架，它将历史经验重构为潜在记忆标记，并直接将其与VLA推理交织融合。LaMem-VLA的核心包含四个协同组件：（i）策展器，将历史经验组织为互补的短期与长期记忆库；（ii）查询器，利用多模态认知从两个记忆库中检索与上下文相关的证据；（iii）压缩器，将检索到的证据重构为紧凑的短期和长期潜在记忆标记；（iv）编织器，将这些记忆标记与当前观测及指令共同注入连续的嵌入序列。通过在统一连续潜在空间中表示、检索和利用历史经验，LaMem-VLA使记忆能直接参与VLA推理，并在有限上下文约束下引导动作生成。在SimplerEnv和LIBERO上的大量实验证明了LaMem-VLA的优越性。

English

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.