DreamCar：利用车辆特定先验知识进行野外环境下的3D汽车重建

DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

July 24, 2024

作者: Xiaobiao Du, Haiyang Sun, Ming Lu, Tianqing Zhu, Xin Yu

cs.AI

摘要

自动驾驶行业通常雇佣专业艺术家来构建精美的3D汽车。然而，制作大规模数字资产成本高昂。由于已经有许多包含大量汽车图像的数据集可用，我们专注于从这些数据集中重建高质量的3D汽车模型。然而，这些数据集只包含前进场景中汽车的一侧。我们尝试利用现有的生成模型提供更多监督信息，但由于它们是在合成数据集而非特定于汽车的数据集上训练的，因此在汽车方面很难泛化。此外，处理野外图像时，由于相机姿态估计存在较大误差，重建的3D汽车纹理会出现错位。这些限制使先前的方法难以重建完整的3D汽车。为了解决这些问题，我们提出了一种名为DreamCar的新方法，可以根据少量甚至单个图像重建高质量的3D汽车。为了泛化生成模型，我们收集了一个名为Car360的汽车数据集，其中包含超过5,600辆车辆。借助这个数据集，我们使生成模型对汽车更加健壮。我们利用这个特定于汽车的生成先验来通过得分蒸馏采样指导其重建。为了进一步补充监督信息，我们利用汽车的几何和外观对称性。最后，我们提出了一种姿态优化方法，纠正姿态以解决纹理错位问题。大量实验证明，我们的方法在重建高质量的3D汽车方面明显优于现有方法。我们的代码可在https://xiaobiaodu.github.io/dreamcar-project/找到。

English

Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. https://xiaobiaodu.github.io/dreamcar-project/{Our code is available.}