LucidDreamer：無領域限制的3D高斯潑濺場景生成技術

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

November 22, 2023

作者: Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee

cs.AI

摘要

隨著VR裝置與內容的廣泛應用，對3D場景生成技術的需求日益增長。然而，現有的3D場景生成模型因採用與現實世界差距較大的3D掃描數據集進行訓練，導致其生成場景侷限於特定領域。為突破此限制，我們提出LucidDreamer——一種透過充分發揮現有大規模擴散生成模型優勢的無領域限制場景生成流程。該流程包含「夢境生成」與「對齊融合」兩個交替步驟：首先，為從輸入生成多視角一致圖像，我們將點雲設為每張圖像生成的幾何指引，具體是將部分點雲投影至目標視角，並以此投影作為生成模型進行修復繪製的引導。隨後將修復後的圖像與估計深度圖共同提升至3D空間，構建新點雲。其次，為將新點雲整合至3D場景，我們提出一種對齊算法，能協調融合新生成3D場景的各個部分。最終獲得的3D場景將作為高斯潑濺優化的初始點雲。相較於傳統3D場景生成方法，LucidDreamer無需限定目標場景領域，且能產生細節更豐富的高斯潑濺表示。

English

With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.