DreamScene360：使用全景高斯飛濺進行無限制的文本到3D場景生成

DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

April 10, 2024

作者: Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj, Suya You, Zhangyang Wang, Achuta Kadambi

cs.AI

摘要

隨著對虛擬實境應用的需求不斷增加，突顯了打造身臨其境的3D資產的重要性。我們提出了一種文本轉3D 360度場景生成流程，有助於在野外環境中快速創建全面的360度場景。我們的方法利用2D擴散模型的生成能力和即時自我精煉，創建高質量且全局一致的全景圖像。這個圖像作為初步的“平面”（2D）場景表示。隨後，通過將其提升為3D高斯模型，採用點陣技術實現實時探索。為了生成一致的3D幾何結構，我們的流程通過將2D單眼深度對齊為全局優化的點雲，構建了一個空間一致的結構。這個點雲作為3D高斯模型的中心的初始狀態。為了解決單視圖輸入固有的不可見問題，我們對合成和輸入相機視圖施加語義和幾何約束作為規範。這些約束指導高斯模型的優化，有助於重建不可見區域。總之，我們的方法提供了一個全局一致的360度視角內的3D場景，相較於現有技術，提供了更加豐富的身臨其境體驗。項目網站：http://dreamscene360.github.io/

English

The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360^{circ} scene generation pipeline that facilitates the creation of comprehensive 360^{circ} scenes for in-the-wild environments in a matter of minutes. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement to create a high-quality and globally coherent panoramic image. This image acts as a preliminary "flat" (2D) scene representation. Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to enable real-time exploration. To produce consistent 3D geometry, our pipeline constructs a spatially coherent structure by aligning the 2D monocular depth into a globally optimized point cloud. This point cloud serves as the initial state for the centroids of 3D Gaussians. In order to address invisible issues inherent in single-view inputs, we impose semantic and geometric constraints on both synthesized and input camera views as regularizations. These guide the optimization of Gaussians, aiding in the reconstruction of unseen regions. In summary, our method offers a globally consistent 3D scene within a 360^{circ} perspective, providing an enhanced immersive experience over existing techniques. Project website at: http://dreamscene360.github.io/