4K4DGen：以4K解析度生成全景4D影像

4K4DGen: Panoramic 4D Generation at 4K Resolution

June 19, 2024

作者: Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhiwen Fan

cs.AI

摘要

虛擬實境和擴增實境（VR/AR）技術的蓬勃發展推動了對高質量、身臨其境且動態環境創建的需求不斷增加。然而，現有的生成技術要麼僅專注於動態物體，要麼從單一視角圖像進行外部繪製，未能滿足VR/AR應用的需求。在這項工作中，我們致力於將單一全景提升為身臨其境的4D體驗這一具有挑戰性的任務。我們首次展示了生成具有360度視角的全方位動態場景，解析度為4K，從而提供身臨其境的用戶體驗。我們的方法引入了一個流程，促進自然場景動畫並使用高效的點狀技術優化一組4D高斯函數，以進行實時探索。為了克服在全景格式中缺乏場景尺度標註的4D數據和模型，我們提出了一種新穎的全景去噪器，將通用的2D擴散先驗適應到360度圖像中，實現一致的動畫，將其轉換為在目標區域具有動態場景的全景視頻。隨後，我們將全景視頻提升為4D身臨其境環境，同時保持空間和時間的一致性。通過將透視域中的2D模型的先前知識轉移到全景域和具有空間外觀和幾何正則化的4D提升，我們首次實現了高質量的全景到4D生成，解析度為（4096乘以2048）。請查看項目網站：https://4k4dgen.github.io。

English

The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the needs of VR/AR applications. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of 4D Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360-degree images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of (4096 times 2048) for the first time. See the project website at https://4k4dgen.github.io.