动态城市：从动态场景生成大规模LiDAR数据

DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes

October 23, 2024

作者: Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei Liu

cs.AI

摘要

最近，LiDAR场景生成技术发展迅速。然而，现有方法主要集中在生成静态和单帧场景，忽视了现实世界驾驶环境固有的动态特性。在这项工作中，我们介绍了DynamicCity，这是一种新颖的4D LiDAR生成框架，能够生成大规模、高质量的LiDAR场景，捕捉动态环境的时间演变。DynamicCity主要由两个关键模型组成。1）VAE模型用于学习HexPlane作为紧凑的4D表示。DynamicCity采用一种新颖的Projection Module，而不是使用简单的平均操作，能够有效地将4D LiDAR特征压缩成六个2D特征图，用于HexPlane构建，从而显著提高HexPlane的拟合质量（最高可达12.56的mIoU增益）。此外，我们利用扩展与压缩策略并行重构3D特征体积，比起简单地查询每个3D点，这既提高了网络训练效率，也提高了重构精度（最高可达7.05的mIoU增益，2.06倍的训练加速度，以及70.84%的内存减少）。2）基于DiT的扩散模型用于HexPlane生成。为了使HexPlane适用于DiT生成，提出了一种填充滚动操作，以将HexPlane的所有六个特征平面重新组织为一个方形的2D特征图。特别是，在扩散或采样过程中可以引入各种条件，支持多样化的4D生成应用，如轨迹驱动和命令驱动生成，修复，以及布局条件生成。对CarlaSC和Waymo数据集进行的大量实验表明，DynamicCity在多个指标上显著优于现有的最先进的4D LiDAR生成方法。我们将发布代码以促进未来的研究。

English

LiDAR scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework capable of generating large-scale, high-quality LiDAR scenes that capture the temporal evolution of dynamic environments. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D LiDAR features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D LiDAR generation methods across multiple metrics. The code will be released to facilitate future research.