SpaRP：從稀疏視角快速進行3D物體重建與姿態估計

SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views

August 19, 2024

作者: Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, Minghua Liu

cs.AI

摘要

最近，開放世界的3D生成引起了相當大的關注。雖然許多單圖像到3D的方法產生了視覺上吸引人的結果，但它們通常缺乏足夠的可控性，並且往往會產生幻覺區域，這些區域可能與用戶的期望不符。在本文中，我們探索了一個重要的情境，其中輸入包括一個或幾個未擺姿勢的單個物體的2D圖像，幾乎沒有重疊。我們提出了一種新的方法，名為SpaRP，用於重建一個帶紋理的3D網格並估計這些稀疏視圖的相對相機姿勢。SpaRP從2D擴散模型中提煉知識，並對其進行微調，以隱含地推斷稀疏視圖之間的3D空間關係。擴散模型被訓練來共同預測相機姿勢的替代表示和對象在已知姿勢下的多視圖圖像，整合來自輸入稀疏視圖的所有信息。然後利用這些預測來完成3D重建和姿勢估計，並且重建的3D模型可以用來進一步優化輸入視圖的相機姿勢。通過對三個數據集進行廣泛實驗，我們展示了我們的方法不僅在3D重建質量和姿勢預測準確性方面顯著優於基線方法，而且表現出強大的效率。它僅需要約20秒的時間來為輸入視圖生成帶紋理的網格和相機姿勢。項目頁面：https://chaoxu.xyz/sparp。

English

Open-world 3D generation has recently attracted considerable attention. While many single-image-to-3D methods have yielded visually appealing outcomes, they often lack sufficient controllability and tend to produce hallucinated regions that may not align with users' expectations. In this paper, we explore an important scenario in which the input consists of one or a few unposed 2D images of a single object, with little or no overlap. We propose a novel method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative camera poses for these sparse-view images. SpaRP distills knowledge from 2D diffusion models and finetunes them to implicitly deduce the 3D spatial relationships between the sparse views. The diffusion model is trained to jointly predict surrogate representations for camera poses and multi-view images of the object under known poses, integrating all information from the input sparse views. These predictions are then leveraged to accomplish 3D reconstruction and pose estimation, and the reconstructed 3D model can be used to further refine the camera poses of input views. Through extensive experiments on three datasets, we demonstrate that our method not only significantly outperforms baseline methods in terms of 3D reconstruction quality and pose prediction accuracy but also exhibits strong efficiency. It requires only about 20 seconds to produce a textured mesh and camera poses for the input views. Project page: https://chaoxu.xyz/sparp.