SPEAR：一個用於照片級真實感具身AI研究的模擬器

摘要

互動式模擬器已成為訓練具身智慧體與生成合成視覺數據的強大工具，但現有照片級真實感模擬器普遍存在通用性不足、可程式化程度有限及渲染速度緩慢等問題。為解決這些限制，我們提出SPEAR：一項專為照片級真實感具身AI研究所設計的模擬器。其核心為Python函式庫，能透過模組化插件架構連接並程式化控制任何虛幻引擎（UE）應用程式。SPEAR提供超過14,000種UE函數的Python介面，相較現有UE模擬器實現了一個數量級的可程式化功能擴增。此外，單一SPEAR實例能以每秒73幀的速度直接將1920×1080解析度的照片級真實感影像渲染至用戶的NumPy陣列——比現有UE插件快上一個數量級——同時提供現有UE模擬器無法取得的真值影像模態（例如非漫反射本質影像分解、材質識別碼及基於物理的著色參數）。最後，SPEAR引入一套具表達力的高階程式設計模型，能讓使用者定義包含任意資料相依性的UE工作複雜圖形，並在單一UE幀內確定性地執行這些圖形。我們透過多樣化的應用實例展示SPEAR的實用性：在數個即時UE專案中控制多個具獨特動作空間的具身智慧體（如人類、車輛與機器人）；渲染照片級真實感的城市規模環境；操控UE的程式化內容生成系統；渲染高清人臉的同步多視角影像；協調與MuJoCo物理模擬器的互動式協同模擬；以及透過AI程式碼輔助系統以自然語言編輯場景。

English

Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.