單張圖像的三維高斯編輯

3D Gaussian Editing with A Single Image

August 14, 2024

作者: Guan Luo, Tian-Xing Xu, Ying-Tian Liu, Xiao-Xiong Fan, Fang-Lue Zhang, Song-Hai Zhang

cs.AI

摘要

從現實世界捕捉的3D場景的建模和操作在各種應用中至關重要，吸引了越來越多的研究興趣。儘管先前關於編輯的研究通過操作3D網格取得了有趣的結果，但它們通常需要精確重建的網格來執行編輯，這限制了它們在3D內容生成中的應用。為了填補這一差距，我們引入了一種基於3D高斯飄落的新型單圖驅動3D場景編輯方法，通過直接在2D圖像平面上編輯內容實現直觀操作。我們的方法學習優化3D高斯分佈，使其與從原始場景的用戶指定視角渲染的圖像的編輯版本對齊。為了捕捉長程物體變形，我們在3D高斯飄落的優化過程中引入位置損失，並通過重新參數化實現梯度傳播。為了處理從指定視角渲染時的遮蔽3D高斯分佈，我們構建了一個基於錨點的結構，並採用了粗到細的優化策略，能夠處理長程變形同時保持結構穩定性。此外，我們設計了一種新穎的遮罩策略，以自適應方式識別非剛性變形區域進行細節建模。大量實驗表明，我們的方法在處理幾何細節、長程和非剛性變形方面的有效性，展示了與先前方法相比更優的編輯靈活性和質量。

English

The modeling and manipulation of 3D scenes captured from the real world are pivotal in various applications, attracting growing research interest. While previous works on editing have achieved interesting results through manipulating 3D meshes, they often require accurately reconstructed meshes to perform editing, which limits their application in 3D content generation. To address this gap, we introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint of the original scene. To capture long-range object deformation, we introduce positional loss into the optimization process of 3D Gaussian Splatting and enable gradient propagation through reparameterization. To handle occluded 3D Gaussians when rendering from the specified viewpoint, we build an anchor-based structure and employ a coarse-to-fine optimization strategy capable of handling long-range deformation while maintaining structural stability. Furthermore, we design a novel masking strategy to adaptively identify non-rigid deformation regions for fine-scale modeling. Extensive experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation, demonstrating superior editing flexibility and quality compared to previous approaches.