LayerPano3D：用於超沉浸式場景生成的分層3D全景

LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

August 23, 2024

作者: Shuai Yang, Jing Tan, Mengchen Zhang, Tong Wu, Yixuan Li, Gordon Wetzstein, Ziwei Liu, Dahua Lin

cs.AI

摘要

在計算機視覺和圖形學中，3D沉浸式場景生成是一項具有挑戰性但至關重要的任務。理想的虛擬3D場景應該1）展現全方位視角一致性，並且2）允許在複雜場景層次結構中進行自由探索。現有方法要麼依賴通過修補逐步擴展場景，要麼使用全景表示法來呈現大視場場景環境。然而，在擴展過程中生成的場景容易出現語義漂移，並且無法處理場景層次結構中的遮擋。為應對這些挑戰，我們提出LayerPano3D，這是一個從單個文本提示生成全視角、可探索的全景3D場景的新框架。我們的關鍵洞察是將參考的2D全景圖分解為不同深度層級的多個層，每個層通過擴散先驗從參考視角中顯示未見空間。LayerPano3D包括多個專用設計：1）我們引入了一個新的文本引導錨點視圖合成管道，用於高質量、一致性全景生成。2）我們開創了分層3D全景作為底層表示，以管理複雜的場景層次結構，並將其提升為3D高斯函數，以噴灑詳細的360度全方位場景，並實現無限制的觀看路徑。大量實驗表明，我們的框架在全視角一致性和沉浸式探索體驗方面生成了最先進的3D全景場景。我們相信LayerPano3D有望推動3D全景場景創建，並具有眾多應用潛力。

English

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.