夢立方：基於多平面同步的三維全景生成

摘要

三維全景合成是一項前景廣闊卻又充滿挑戰的任務，它要求生成的環視內容具備高質量且多樣化的視覺外觀與幾何結構。現有方法利用預訓練的二維基礎模型所提供的豐富圖像先驗，以應對三維全景數據稀缺的問題，但三維全景與二維單視圖之間的不兼容性限制了這些方法的效能。在本研究中，我們展示了通過對二維基礎模型中的算子應用多平面同步技術，能夠無縫地將其能力擴展至環視領域。基於這一設計，我們進一步引入了DreamCube，這是一個用於三維全景生成的多平面RGB-D擴散模型，它最大限度地重用了二維基礎模型的先驗知識，以實現多樣化的外觀與精確的幾何結構，同時保持多視圖一致性。大量實驗證明了我們的方法在全景圖像生成、全景深度估計以及三維場景生成中的有效性。

English

3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.

夢立方：基於多平面同步的三維全景生成

DreamCube: 3D Panorama Generation via Multi-plane Synchronization

摘要

Support