擴散增強智能體：高效探索與遷移學習的框架

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

July 30, 2024

作者: Norman Di Palo, Leonard Hasenclever, Jan Humplik, Arunkumar Byravan

cs.AI

摘要

我們介紹了擴散增強代理（Diffusion Augmented Agents，DAAG），這是一個新穎的框架，利用大型語言模型、視覺語言模型和擴散模型，以提高具身體代理的強化學習中的樣本效率和遷移學習。DAAG透過擴散模型將代理的過去經驗進行事後重新標記，以一種我們稱為事後經驗增強的技術，以時間和幾何一致的方式轉換視頻，以與目標指令對齊。一個大型語言模型協調這個自主過程，無需人類監督，非常適合終身學習場景。該框架減少了需要標記獎勵數據的量，以便1）微調作為獎勵檢測器的視覺語言模型，和2）對新任務訓練強化學習代理。我們展示了DAAG在涉及機器人操作和導航的模擬環境中的樣本效率增益。我們的結果顯示，DAAG改善了獎勵檢測器的學習、遷移過去經驗，以及獲取新任務的能力 - 這是發展高效終身學習代理的關鍵能力。我們的網站提供了補充資料和視覺化：https://sites.google.com/view/diffusion-augmented-agents/

English

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents. DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation. A large language model orchestrates this autonomous process without requiring human supervision, making it well-suited for lifelong learning scenarios. The framework reduces the amount of reward-labeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks. We demonstrate the sample efficiency gains of DAAG in simulated robotics environments involving manipulation and navigation. Our results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks - key abilities for developing efficient lifelong learning agents. Supplementary material and visualizations are available on our website https://sites.google.com/view/diffusion-augmented-agents/