利用单个图像进行3D高斯编辑

3D Gaussian Editing with A Single Image

August 14, 2024

作者: Guan Luo, Tian-Xing Xu, Ying-Tian Liu, Xiao-Xiong Fan, Fang-Lue Zhang, Song-Hai Zhang

cs.AI

摘要

从现实世界捕获的3D场景的建模和操作在各种应用中至关重要，吸引了越来越多的研究兴趣。虽然先前关于编辑的研究通过操纵3D网格取得了有趣的结果，但它们通常需要准确重建的网格来执行编辑，这限制了它们在3D内容生成中的应用。为了填补这一空白，我们引入了一种基于3D高斯散射的新型单图驱动3D场景编辑方法，通过直接在2D图像平面上编辑内容实现直观操作。我们的方法学习优化3D高斯函数，使其与从原始场景的用户指定视点渲染的图像的编辑版本对齐。为了捕捉远程物体变形，我们在3D高斯散射的优化过程中引入位置损失，并通过重新参数化实现梯度传播。为了处理从指定视点渲染时的遮挡3D高斯函数，我们构建了一个基于锚点的结构，并采用了适用于处理远程变形并保持结构稳定的粗到细的优化策略。此外，我们设计了一种新颖的遮罩策略，自适应地识别非刚性变形区域进行精细建模。大量实验证明了我们的方法在处理几何细节、远程和非刚性变形方面的有效性，展示了与先前方法相比卓越的编辑灵活性和质量。

English

The modeling and manipulation of 3D scenes captured from the real world are pivotal in various applications, attracting growing research interest. While previous works on editing have achieved interesting results through manipulating 3D meshes, they often require accurately reconstructed meshes to perform editing, which limits their application in 3D content generation. To address this gap, we introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting, enabling intuitive manipulation via directly editing the content on a 2D image plane. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint of the original scene. To capture long-range object deformation, we introduce positional loss into the optimization process of 3D Gaussian Splatting and enable gradient propagation through reparameterization. To handle occluded 3D Gaussians when rendering from the specified viewpoint, we build an anchor-based structure and employ a coarse-to-fine optimization strategy capable of handling long-range deformation while maintaining structural stability. Furthermore, we design a novel masking strategy to adaptively identify non-rigid deformation regions for fine-scale modeling. Extensive experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation, demonstrating superior editing flexibility and quality compared to previous approaches.