SyntheOcc：通过3D语义MPIs合成几何控制的街景图像

摘要

自动驾驶技术的进步越来越依赖于高质量的标注数据集，特别是在3D占据预测任务中，占据标签需要密集的3D标注，需要大量人力投入。在本文中，我们提出了SyntheOcc，这是一个表示扩散模型，通过将驾驶场景中的占据标签作为条件，合成具有照片逼真性和几何控制的图像。这为训练感知模型和模拟等应用提供了数量不限、多样化、可控的数据集。SyntheOcc解决了如何有效地将3D几何信息编码为2D扩散模型的条件输入的关键挑战。我们的方法创新地将3D语义多平面图像（MPIs）结合起来，为条件输入提供全面且空间对齐的3D场景描述。因此，SyntheOcc能够生成与给定几何标签（3D体素空间中的语义）完全对齐的照片逼真的多视角图像和视频。对nuScenes数据集上对SyntheOcc进行的广泛定性和定量评估证明了它在生成可控占据数据集方面的有效性，可作为感知模型的有效数据增强。

English

The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.

SyntheOcc：通过3D语义MPIs合成几何控制的街景图像

SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

摘要

Support