基于生成扩散先验的分解式神经场景重建

Decompositional Neural Scene Reconstruction with Generative Diffusion Prior

March 19, 2025

作者: Junfeng Ni, Yu Liu, Ruijie Lu, Zirui Zhou, Song-Chun Zhu, Yixin Chen, Siyuan Huang

cs.AI

摘要

三维场景的分解式重建，包含所有物体的完整形状与精细纹理，对于下游应用极具吸引力，但在稀疏视角输入条件下仍面临挑战。近期方法通过引入语义或几何正则化来应对此问题，但在约束不足区域表现显著下降，且难以恢复被遮挡部分。我们认为，解决这一难题的关键在于为这些区域补充缺失信息。为此，我们提出了DP-Recon，它利用扩散先验，以分数蒸馏采样（SDS）的形式，优化新视角下每个独立物体的神经表示。这为约束不足区域提供了额外信息，但直接引入扩散先验可能导致重建与生成指导之间的潜在冲突。因此，我们进一步提出了一种基于可见性的方法，动态调整逐像素SDS损失权重。这些组件共同作用，在保持对输入图像忠实度的同时，提升了几何与外观的恢复效果。在Replica和ScanNet++数据集上的广泛实验表明，我们的方法显著超越了现有最先进技术。尤为突出的是，在仅10个视角下，我们的方法就能实现比基线方法在100个视角下更优的物体重建。通过SDS优化，我们的方法支持基于文本的几何与外观无缝编辑，并生成带有详细UV贴图的分解物体网格，为逼真的视觉特效（VFX）编辑提供了支持。项目页面详见https://dp-recon.github.io/。

English

Decompositional reconstruction of 3D scenes, with complete shapes and detailed texture of all objects within, is intriguing for downstream applications but remains challenging, particularly with sparse views as input. Recent approaches incorporate semantic or geometric regularization to address this issue, but they suffer significant degradation in underconstrained areas and fail to recover occluded regions. We argue that the key to solving this problem lies in supplementing missing information for these areas. To this end, we propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views. This provides additional information for the underconstrained areas, but directly incorporating diffusion prior raises potential conflicts between the reconstruction and generative guidance. Therefore, we further introduce a visibility-guided approach to dynamically adjust the per-pixel SDS loss weights. Together these components enhance both geometry and appearance recovery while remaining faithful to input images. Extensive experiments across Replica and ScanNet++ demonstrate that our method significantly outperforms SOTA methods. Notably, it achieves better object reconstruction under 10 views than the baselines under 100 views. Our method enables seamless text-based editing for geometry and appearance through SDS optimization and produces decomposed object meshes with detailed UV maps that support photorealistic Visual effects (VFX) editing. The project page is available at https://dp-recon.github.io/.