DreamSpace：使用基於文字驅動的全景紋理擴散來夢想您的房間空間
DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation
October 19, 2023
作者: Bangbang Yang, Wenqi Dong, Lin Ma, Wenbo Hu, Xiao Liu, Zhaopeng Cui, Yuewen Ma
cs.AI
摘要
基於擴散的方法在生成2D媒體方面取得了顯著成功。然而，在3D空間應用中實現類似的場景級網格紋理技能，例如XR/VR，仍受限，主要是由於3D幾何的複雜性以及對沉浸式自由視點渲染的需求。本文提出了一種新穎的室內場景紋理框架，該框架提供了具有迷人細節和真實空間一致性的文本驅動紋理生成。關鍵見解在於首先從場景的中心視點想像出一個風格化的360度全景紋理，然後通過修補和模仿技術將其傳播到其他區域。為了確保紋理對場景有意義且對齊，我們開發了一種新穎的從粗糙到精細的全景紋理生成方法，具有雙紋理對齊，同時考慮了捕捉場景的幾何和紋理線索。為了在紋理傳播過程中避免混亂的幾何結構，我們設計了一種分離策略，該策略在機密區域進行紋理修補，然後學習一個隱式模仿網絡，在遮蔽和微小結構區域合成紋理。對真實世界室內場景進行的大量實驗和沉浸式VR應用展示了生成紋理的高質量以及在VR頭戴設備上引人入勝的體驗。項目網頁：https://ybbbbt.com/publication/dreamspace
English
Diffusion-based methods have achieved prominent success in generating 2D
media. However, accomplishing similar proficiencies for scene-level mesh
texturing in 3D spatial applications, e.g., XR/VR, remains constrained,
primarily due to the intricate nature of 3D geometry and the necessity for
immersive free-viewpoint rendering. In this paper, we propose a novel indoor
scene texturing framework, which delivers text-driven texture generation with
enchanting details and authentic spatial coherence. The key insight is to first
imagine a stylized 360{\deg} panoramic texture from the central viewpoint of
the scene, and then propagate it to the rest areas with inpainting and
imitating techniques. To ensure meaningful and aligned textures to the scene,
we develop a novel coarse-to-fine panoramic texture generation approach with
dual texture alignment, which both considers the geometry and texture cues of
the captured scenes. To survive from cluttered geometries during texture
propagation, we design a separated strategy, which conducts texture inpainting
in confidential regions and then learns an implicit imitating network to
synthesize textures in occluded and tiny structural areas. Extensive
experiments and the immersive VR application on real-world indoor scenes
demonstrate the high quality of the generated textures and the engaging
experience on VR headsets. Project webpage:
https://ybbbbt.com/publication/dreamspace