基於生成擴散先驗的分解式神經場景重建

Decompositional Neural Scene Reconstruction with Generative Diffusion Prior

March 19, 2025

作者: Junfeng Ni, Yu Liu, Ruijie Lu, Zirui Zhou, Song-Chun Zhu, Yixin Chen, Siyuan Huang

cs.AI

摘要

三維場景的分解重建，包含所有物體的完整形狀和細緻紋理，對於下游應用極具吸引力，但在稀疏視角輸入的情況下仍面臨挑戰。現有方法通過引入語義或幾何正則化來解決這一問題，但在約束不足的區域會出現顯著退化，且無法恢復被遮擋的區域。我們認為，解決這一問題的關鍵在於為這些區域補充缺失的信息。為此，我們提出了DP-Recon，該方法利用擴散先驗，以分數蒸餾採樣（SDS）的形式，在新視角下優化每個單獨物體的神經表示。這為約束不足的區域提供了額外信息，但直接引入擴散先驗可能會導致重建與生成指導之間的潛在衝突。因此，我們進一步引入了一種基於可見性的方法，動態調整每個像素的SDS損失權重。這些組件共同增強了幾何和外觀的恢復，同時忠實於輸入圖像。在Replica和ScanNet++上的大量實驗表明，我們的方法顯著優於現有的SOTA方法。值得注意的是，在10個視角下，我們的物體重建效果優於基線方法在100個視角下的表現。我們的方法通過SDS優化實現了基於文本的幾何和外觀無縫編輯，並生成了帶有詳細UV貼圖的分解物體網格，支持照片級真實感的視覺特效（VFX）編輯。項目頁面可在https://dp-recon.github.io/ 訪問。

English

Decompositional reconstruction of 3D scenes, with complete shapes and detailed texture of all objects within, is intriguing for downstream applications but remains challenging, particularly with sparse views as input. Recent approaches incorporate semantic or geometric regularization to address this issue, but they suffer significant degradation in underconstrained areas and fail to recover occluded regions. We argue that the key to solving this problem lies in supplementing missing information for these areas. To this end, we propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views. This provides additional information for the underconstrained areas, but directly incorporating diffusion prior raises potential conflicts between the reconstruction and generative guidance. Therefore, we further introduce a visibility-guided approach to dynamically adjust the per-pixel SDS loss weights. Together these components enhance both geometry and appearance recovery while remaining faithful to input images. Extensive experiments across Replica and ScanNet++ demonstrate that our method significantly outperforms SOTA methods. Notably, it achieves better object reconstruction under 10 views than the baselines under 100 views. Our method enables seamless text-based editing for geometry and appearance through SDS optimization and produces decomposed object meshes with detailed UV maps that support photorealistic Visual effects (VFX) editing. The project page is available at https://dp-recon.github.io/.