Image2Sim: 通过生成式神经模拟器扩展具身导航

摘要

具身导航旨在构建能够理解多模态目标、在三维空间中推理并在现实世界中可靠地到达目的地的智能体。然而，由于缺乏可拓展、高保真度且具备物理真实感的交互环境，该领域的进展仍受到制约。尽管真实世界扫描数据集提供了视觉真实感，但其规模有限。相比之下，合成仿真器虽更易拓展，却常存在较大的仿真到现实差距。我们提出Image2Sim——一种实时神经模拟框架，能从带位姿的RGB-D图像序列中构建高质量交互环境。其核心思想是将3D空间锚定与逼真观察合成解耦。在场景构建方面，Image2Sim采用前馈特征高斯模型，通过单次前向传播将带位姿的RGB-D观测提升为3D特征高斯表示。在渲染方面，我们提出几何感知一步像素流模型，能将稀疏且带噪声的高斯投影转化为高质量全景RGB-D观测。此外，Image2Sim还可作为全自动具身数据引擎，大规模生成高保真观测、可执行动作及多样化导航指令。它可将大量视频与图像转换为近2万个交互场景，并合成超过1000万个导航训练样本。完全在这些神经环境中训练的导航模型在主流基准测试上取得显著提升，并能有效迁移至现实世界的零样本场景。这些结果表明，可拓展的神经模拟可成为大规模具身导航的实用训练基础。

English

Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.