Image2Sim：通过生成式神经模拟器扩展具身导航

摘要

具身导航旨在构建能够理解多模态目标、在三维空间中推理并可靠地在现实世界中抵达目标位置的智能体。然而，由于缺乏可扩展、高保真且基于物理交互的环境，该领域的发展仍然受限。尽管现实世界扫描数据集提供了视觉逼真度，但其规模有限。相比之下，合成仿真器更易扩展，但往往存在较大的仿真到现实差距。我们提出Image2Sim，一种实时神经仿真框架，能够从带有姿态的RGB-D图像序列构建高质量的交互环境。其核心思想是将三维空间锚定与逼真观测合成相解耦。在场景构建方面，Image2Sim使用前馈特征高斯模型，将带姿态的RGB-D观测通过单次前向传播提升为三维特征高斯表示。在渲染方面，我们提出几何感知单步像素流模型，将稀疏且带噪声的高斯投影转化为高质量的全景RGB-D观测。Image2Sim还可作为全自动的具身数据引擎，大规模生成高保真观测、可执行动作及多样化导航指令。它能够将大量视频和图像转换为近2万个交互场景，并合成超过1000万个导航训练样本。完全在这些神经环境中训练的导航模型，在主要基准测试上取得了显著改进，并能有效迁移至现实世界的零样本场景。这些结果表明，可扩展的神经仿真可作为大规模具身导航的实用训练基础。

English

Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.