艾拉：具备终身记忆的具身社交智能体

摘要

我们推出Ella，一个能够在3D开放世界中通过终身学习与社区互动的具身社交智能体。Ella通过日常视觉观察和社交互动积累经验、获取知识。其核心能力在于一个结构化的长期多模态记忆系统，该系统高效地存储、更新和检索信息。该系统由以名称为中心的语义记忆（用于组织获取的知识）和时空情景记忆（用于捕捉多模态体验）组成。通过将这一终身记忆系统与基础模型相结合，Ella能够检索相关信息以进行决策、规划日常活动、建立社交关系，并在与开放世界中其他智能体共存的同时自主进化。我们在一个动态的3D开放世界中进行能力导向的评估，15个智能体在此进行多日社交活动，并通过一系列未见过的控制性测试进行评估。实验结果表明，Ella能够有效地影响、领导并与其他智能体合作达成目标，展现了其通过观察和社交互动高效学习的能力。我们的研究结果突显了将结构化记忆系统与基础模型相结合在推进具身智能方面的变革潜力。更多视频请访问https://umass-embodied-agi.github.io/Ella/。

English

We introduce Ella, an embodied social agent capable of lifelong learning within a community in a 3D open world, where agents accumulate experiences and acquire knowledge through everyday visual observations and social interactions. At the core of Ella's capabilities is a structured, long-term multimodal memory system that stores, updates, and retrieves information effectively. It consists of a name-centric semantic memory for organizing acquired knowledge and a spatiotemporal episodic memory for capturing multimodal experiences. By integrating this lifelong memory system with foundation models, Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. We conduct capability-oriented evaluations in a dynamic 3D open world where 15 agents engage in social activities for days and are assessed with a suite of unseen controlled evaluations. Experimental results show that Ella can influence, lead, and cooperate with other agents well to achieve goals, showcasing its ability to learn effectively through observation and social interaction. Our findings highlight the transformative potential of combining structured memory systems with foundation models for advancing embodied intelligence. More videos can be found at https://umass-embodied-agi.github.io/Ella/.

艾拉：具备终身记忆的具身社交智能体

Ella: Embodied Social Agents with Lifelong Memory

摘要

Support