Difix3D+：利用單步擴散模型提升3D重建效果

Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models

March 3, 2025

作者: Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojcic, Huan Ling

cs.AI

摘要

神經輻射場（Neural Radiance Fields）與三維高斯濺射（3D Gaussian Splatting）技術已徹底革新了三維重建與新視角合成任務。然而，從極端新視角實現照片級真實感渲染仍具挑戰性，因為各種表示方法中仍存在偽影。在本研究中，我們提出了Difix3D+，一種旨在通過單步擴散模型提升三維重建與新視角合成質量的新穎流程。我們方法的核心是Difix，這是一個單步圖像擴散模型，專門訓練來增強並去除由三維表示中約束不足區域所導致的渲染新視角中的偽影。Difix在我們的流程中扮演著兩個關鍵角色。首先，在重建階段，它被用於清理從重建結果渲染出的偽訓練視圖，隨後這些視圖被蒸餾回三維空間，這大大增強了約束不足區域並提升了整體三維表示的質量。更重要的是，Difix在推理階段還作為神經增強器，有效去除因不完善的三維監督及當前重建模型能力限制所產生的殘餘偽影。Difix3D+是一個通用解決方案，一個兼容NeRF與3DGS表示的單一模型，它在保持三維一致性的同時，相較於基線模型，FID分數平均提升了2倍。

English

Neural Radiance Fields and 3D Gaussian Splatting have revolutionized 3D reconstruction and novel-view synthesis task. However, achieving photorealistic rendering from extreme novel viewpoints remains challenging, as artifacts persist across representations. In this work, we introduce Difix3D+, a novel pipeline designed to enhance 3D reconstruction and novel-view synthesis through single-step diffusion models. At the core of our approach is Difix, a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by underconstrained regions of the 3D representation. Difix serves two critical roles in our pipeline. First, it is used during the reconstruction phase to clean up pseudo-training views that are rendered from the reconstruction and then distilled back into 3D. This greatly enhances underconstrained regions and improves the overall 3D representation quality. More importantly, Difix also acts as a neural enhancer during inference, effectively removing residual artifacts arising from imperfect 3D supervision and the limited capacity of current reconstruction models. Difix3D+ is a general solution, a single model compatible with both NeRF and 3DGS representations, and it achieves an average 2times improvement in FID score over baselines while maintaining 3D consistency.