记忆增强型视觉语言代理：实现持久且语义一致的目标描述生成

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

March 30, 2026

作者: Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale

cs.AI

摘要

视觉语言模型（VLMs）在对同一物体进行多视角描述时往往存在不一致性，这阻碍了具身智能体构建持续连贯的语义表征能力。现有方法通过离线多视图聚合或多阶段流程（将探索、数据关联与描述学习解耦）来解决不一致问题，但难以对历史观察对象进行推理。本文提出一种统一的记忆增强型视觉语言智能体，在自回归框架内同步处理数据关联、物体描述生成与探索策略。该模型通过处理当前RGB观测数据、自上而下的探索地图，以及序列化为物体级标记的情景记忆，确保在长时序中维持物体身份与语义一致性。为实现自监督训练，我们在逼真3D环境中采用基于分歧的探索策略和伪描述生成模型（通过强化多视角描述历史一致性）构建数据集。经人工标注的物体级测试集验证，本模型在标准描述评分上较基线模型提升最高达11.86%，描述自相似度提升7.39%，同时通过紧凑场景表征实现可扩展性能。代码、模型权重及数据详见https://hsp-iit.github.io/epos-vlm/。

English

Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.