DreamCar：利用針對汽車的先驗知識進行野外3D汽車重建

DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

July 24, 2024

作者: Xiaobiao Du, Haiyang Sun, Ming Lu, Tianqing Zhu, Xin Yu

cs.AI

摘要

自駕車行業通常聘請專業藝術家來建立精美的3D汽車。然而，製作大規模數位資產成本高昂。由於已經有許多包含大量汽車圖像的數據集可用，我們專注於從這些數據集中重建高質量的3D汽車模型。然而，這些數據集僅包含前進場景中汽車的一側。我們嘗試使用現有的生成模型提供更多監督信息，但由於它們是在合成數據集而非汽車特定數據集上訓練的，因此在汽車方面很難很好地泛化。此外，在處理野外圖像時，由於相機姿勢估計存在較大誤差，重建的3D汽車紋理會出現錯位。這些限制使得以前的方法難以重建完整的3D汽車。為解決這些問題，我們提出了一種名為DreamCar的新方法，可以重建高質量的3D汽車，即使只提供少量圖像甚至單張圖像。為了泛化生成模型，我們收集了一個名為Car360的汽車數據集，其中包含超過5,600輛車輛。通過這個數據集，我們使生成模型對汽車更加強健。我們使用這個針對汽車的生成先驗來引導其重建，通過得分蒸餾抽樣。為了進一步補充監督信息，我們利用汽車的幾何和外觀對稱性。最後，我們提出了一種姿勢優化方法，校正姿勢以解決紋理錯位問題。大量實驗表明，我們的方法在重建高質量的3D汽車方面明顯優於現有方法。我們的代碼可在https://xiaobiaodu.github.io/dreamcar-project/找到。

English

Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. https://xiaobiaodu.github.io/dreamcar-project/{Our code is available.}