互動式3D：透過互動式3D生成創建所需物件

Interactive3D: Create What You Want by Interactive 3D Generation

April 25, 2024

作者: Shaocong Dong, Lihe Ding, Zhanpeng Huang, Zibin Wang, Tianfan Xue, Dan Xu

cs.AI

摘要

3D物體生成已經取得了顯著的進展，產生了高質量的結果。然而，缺乏實現精確用戶控制，通常會產生與用戶期望不符的結果，從而限制了其應用範圍。用戶構想的3D物體生成面臨著重大挑戰，現有生成模型由於互動能力有限，難以實現其概念。現有方法主要提供兩種途徑：(i)解釋文本指令，具有受限的可控性，或者(ii)從2D圖像重建3D物體。這兩種方法都將定制限制在2D參考範圍內，並可能在3D轉換過程中引入不良藝術品，限制了直接和多功能的3D修改範圍。在這項工作中，我們介紹了Interactive3D，這是一個創新的互動3D生成框架，通過廣泛的3D互動功能，使用戶對生成過程具有精確控制。Interactive3D由兩個級聯階段構建，利用不同的3D表示。第一階段採用高斯樣板進行直接用戶互動，允許在任何中間步驟通過(i)添加和刪除組件，(ii)可變形和剛性拖動，(iii)幾何變換和(iv)語義編輯來修改和引導生成方向。隨後，高斯樣板被轉換為InstantNGP。我們引入了一個新的(v)互動哈希細化模塊，以進一步添加細節並在第二階段中提取幾何形狀。我們的實驗表明，Interactive3D顯著提高了3D生成的可控性和質量。我們的項目網頁位於https://interactive-3d.github.io/。

English

3D object generation has undergone significant advancements, yielding high-quality results. However, fall short of achieving precise user control, often yielding results that do not align with user expectations, thus limiting their applicability. User-envisioning 3D object generation faces significant challenges in realizing its concepts using current generative models due to limited interaction capabilities. Existing methods mainly offer two approaches: (i) interpreting textual instructions with constrained controllability, or (ii) reconstructing 3D objects from 2D images. Both of them limit customization to the confines of the 2D reference and potentially introduce undesirable artifacts during the 3D lifting process, restricting the scope for direct and versatile 3D modifications. In this work, we introduce Interactive3D, an innovative framework for interactive 3D generation that grants users precise control over the generative process through extensive 3D interaction capabilities. Interactive3D is constructed in two cascading stages, utilizing distinct 3D representations. The first stage employs Gaussian Splatting for direct user interaction, allowing modifications and guidance of the generative direction at any intermediate step through (i) Adding and Removing components, (ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv) Semantic Editing. Subsequently, the Gaussian splats are transformed into InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to further add details and extract the geometry in the second stage. Our experiments demonstrate that Interactive3D markedly improves the controllability and quality of 3D generation. Our project webpage is available at https://interactive-3d.github.io/.