清晰夢想家：無領域生成3D高斯飛濺場景

摘要

隨著虛擬實境設備和內容的廣泛應用，對於3D場景生成技術的需求變得更加普遍。然而，現有的3D場景生成模型限制了目標場景的特定領域，主要是由於它們使用與現實世界相去甚遠的3D掃描數據集進行訓練策略。為了解決這一限制，我們提出了LucidDreamer，這是一個無域場景生成流程，充分利用現有大規模擴散式生成模型的強大功能。我們的LucidDreamer包含兩個交替步驟：夢境和對齊。首先，為了從輸入生成多視角一致的圖像，我們將點雲設置為每個圖像生成的幾何指南。具體來說，我們將點雲的一部分投影到所需的視角，並將該投影作為生成模型進行修補時的指導。修補後的圖像通過估計的深度圖提升到3D空間，形成新的點。其次，為了將新的點聚合到3D場景中，我們提出了一種對齊算法，將新生成的3D場景部分和諧地整合在一起。最終獲得的3D場景作為優化高斯斑點的初始點。與先前的3D場景生成方法相比，LucidDreamer生成的高斯斑點更加細緻，並且不限制目標場景的領域。

English

With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.

清晰夢想家：無領域生成3D高斯飛濺場景

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

摘要

Support