交互式三维：通过交互式三维生成创造您想要的内容

Interactive3D: Create What You Want by Interactive 3D Generation

April 25, 2024

作者: Shaocong Dong, Lihe Ding, Zhanpeng Huang, Zibin Wang, Tianfan Xue, Dan Xu

cs.AI

摘要

3D物体生成已经取得了显著的进展，产生了高质量的结果。然而，它们在实现精确用户控制方面还存在不足，通常会产生与用户期望不符的结果，从而限制了它们的适用性。用户设想的3D物体生成面临着重大挑战，因为当前生成模型的交互能力有限，难以实现其概念。现有方法主要提供两种途径：(i) 解释文本指令并具有受限的可控性，或者(ii) 从2D图像重建3D物体。这两种方法都将定制限制在2D参考范围内，并在3D提升过程中可能引入不良伪影，限制了直接和多样化的3D修改范围。在这项工作中，我们介绍了Interactive3D，这是一个创新的交互式3D生成框架，通过广泛的3D交互能力赋予用户对生成过程的精确控制。Interactive3D分为两个级联阶段，利用不同的3D表示。第一阶段采用高斯点阵化进行直接用户交互，允许在任何中间步骤通过(i) 添加和删除组件，(ii) 可变形和刚性拖动，(iii) 几何变换和(iv) 语义编辑来修改和引导生成方向。随后，高斯点阵被转换为InstantNGP。我们引入了一个新颖的(v) 交互式哈希细化模块，以在第二阶段进一步添加细节并提取几何形状。我们的实验表明，Interactive3D显著改善了3D生成的可控性和质量。我们的项目网页可在https://interactive-3d.github.io/ 上找到。

English

3D object generation has undergone significant advancements, yielding high-quality results. However, fall short of achieving precise user control, often yielding results that do not align with user expectations, thus limiting their applicability. User-envisioning 3D object generation faces significant challenges in realizing its concepts using current generative models due to limited interaction capabilities. Existing methods mainly offer two approaches: (i) interpreting textual instructions with constrained controllability, or (ii) reconstructing 3D objects from 2D images. Both of them limit customization to the confines of the 2D reference and potentially introduce undesirable artifacts during the 3D lifting process, restricting the scope for direct and versatile 3D modifications. In this work, we introduce Interactive3D, an innovative framework for interactive 3D generation that grants users precise control over the generative process through extensive 3D interaction capabilities. Interactive3D is constructed in two cascading stages, utilizing distinct 3D representations. The first stage employs Gaussian Splatting for direct user interaction, allowing modifications and guidance of the generative direction at any intermediate step through (i) Adding and Removing components, (ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv) Semantic Editing. Subsequently, the Gaussian splats are transformed into InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to further add details and extract the geometry in the second stage. Our experiments demonstrate that Interactive3D markedly improves the controllability and quality of 3D generation. Our project webpage is available at https://interactive-3d.github.io/.