SPEAR：面向逼真具身人工智能研究的模拟器

摘要

交互式模拟器已成为训练具身智能体和生成合成视觉数据的强大工具，但现有照片级真实感模拟器存在通用性、可编程性和渲染速度方面的局限。为解决这些局限，我们提出SPEAR：一款面向逼真具身人工智能研究的模拟器。SPEAR的核心是一个Python库，可通过模块化插件架构连接并程序化控制任意虚幻引擎（UE）应用。SPEAR向Python暴露了超过1.4万个独特的UE函数，使其可编程功能数量较现有基于UE的模拟器提升了一个数量级。此外，单个SPEAR实例能以每秒73帧的速度将1920×1080分辨率的逼真美感图像直接渲染到用户的NumPy数组中——速度比现有UE插件快一个数量级——同时还能提供现有基于UE的模拟器无法获取的真值图像模态（例如非漫反射内在图像分解、材质ID及基于物理的着色参数）。最后，SPEAR引入了一种表达力强的高级编程模型，使用户能够指定包含任意数据依赖关系的复杂UE工作图，并在单个UE帧内确定性执行这些图。我们通过一系列多样化的示例应用展示了SPEAR的实用性：在多个野外UE项目中控制具有不同动作空间（如人物、汽车和机器人）的多个具身智能体；渲染逼真的城市规模环境；操控UE的流程化内容生成系统；渲染高保真人脸的多视角同步图像；与MuJoCo物理模拟器协调实现交互式协同仿真；以及通过AI编码助手对场景进行自然语言编辑。

English

Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.