AuraFusion360：擴增未見區域對基於參考的360°無界場景修補進行對齊

AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting

February 7, 2025

作者: Chung-Ho Wu, Yang-Jung Chen, Ying-Huan Chen, Jie-Ying Lee, Bo-Hsu Ke, Chun-Wei Tuan Mu, Yi-Chuan Huang, Chin-Yang Lin, Min-Hung Chen, Yen-Yu Lin, Yu-Lun Liu

cs.AI

摘要

三維場景修補對於從虛擬現實到建築可視化的應用至關重要，然而現有方法在360度無邊界場景中的視角一致性和幾何準確性方面存在困難。我們提出了AuraFusion360，一種新穎的基於參考的方法，可以在由高斯Splatting表示的3D場景中實現高質量的對象去除和孔填充。我們的方法引入了(1) 基於深度感知的未見遮罩生成，用於準確識別遮擋，(2) 自適應引導深度擴散，一種零樣本方法，用於準確的初始點放置，無需額外訓練，以及(3) 基於SDEdit的細節增強，用於多視角一致性。我們還介紹了360-USID，這是第一個針對360度無邊界場景修補的全面數據集，具有地面真實性。大量實驗表明，AuraFusion360明顯優於現有方法，在保持幾何準確性的同時實現了卓越的感知質量，跨劇烈視角變化。請查看我們的項目頁面以獲取視頻結果和數據集，網址為https://kkennethwu.github.io/aurafusion360/。

English

Three-dimensional scene inpainting is crucial for applications from virtual reality to architectural visualization, yet existing methods struggle with view consistency and geometric accuracy in 360{\deg} unbounded scenes. We present AuraFusion360, a novel reference-based method that enables high-quality object removal and hole filling in 3D scenes represented by Gaussian Splatting. Our approach introduces (1) depth-aware unseen mask generation for accurate occlusion identification, (2) Adaptive Guided Depth Diffusion, a zero-shot method for accurate initial point placement without requiring additional training, and (3) SDEdit-based detail enhancement for multi-view coherence. We also introduce 360-USID, the first comprehensive dataset for 360{\deg} unbounded scene inpainting with ground truth. Extensive experiments demonstrate that AuraFusion360 significantly outperforms existing methods, achieving superior perceptual quality while maintaining geometric accuracy across dramatic viewpoint changes. See our project page for video results and the dataset at https://kkennethwu.github.io/aurafusion360/.