TexGen：使用多視角取樣和重新取樣的文本引導3D紋理生成

TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling

August 2, 2024

作者: Dong Huo, Zixin Guo, Xinxin Zuo, Zhihao Shi, Juwei Lu, Peng Dai, Songcen Xu, Li Cheng, Yee-Hong Yang

cs.AI

摘要

針對一個3D網格，我們的目標是合成對應於任意文本描述的3D紋理。目前用於從採樣視圖生成和組合紋理的方法通常會導致明顯的接縫或過度平滑。為了應對這些問題，我們提出了TexGen，這是一個新穎的多視圖採樣和重採樣框架，用於紋理生成，利用了一個預先訓練的文本到圖像擴散模型。為了實現視圖一致的採樣，首先我們在RGB空間中維護一個由去噪步驟參數化的紋理映射，並在每個擴散模型的採樣步驟之後更新，逐步減少視圖差異。利用一種基於注意力的多視圖採樣策略，來在視圖之間廣播外觀信息。為了保留紋理細節，我們開發了一種噪聲重採樣技術，有助於估計噪聲，生成用於後續去噪步驟的輸入，根據文本提示和當前紋理映射的指導。通過大量的定性和定量評估，我們展示了我們提出的方法為具有高度視圖一致性和豐富外觀細節的多樣3D物體產生顯著更好的紋理質量，勝過目前的最先進方法。此外，我們提出的紋理生成技術還可以應用於紋理編輯，同時保留原始身份。更多實驗結果可在https://dong-huo.github.io/TexGen/查看。

English

Given a 3D mesh, we aim to synthesize 3D textures that correspond to arbitrary textual descriptions. Current methods for generating and assembling textures from sampled views often result in prominent seams or excessive smoothing. To tackle these issues, we present TexGen, a novel multi-view sampling and resampling framework for texture generation leveraging a pre-trained text-to-image diffusion model. For view consistent sampling, first of all we maintain a texture map in RGB space that is parameterized by the denoising step and updated after each sampling step of the diffusion model to progressively reduce the view discrepancy. An attention-guided multi-view sampling strategy is exploited to broadcast the appearance information across views. To preserve texture details, we develop a noise resampling technique that aids in the estimation of noise, generating inputs for subsequent denoising steps, as directed by the text prompt and current texture map. Through an extensive amount of qualitative and quantitative evaluations, we demonstrate that our proposed method produces significantly better texture quality for diverse 3D objects with a high degree of view consistency and rich appearance details, outperforming current state-of-the-art methods. Furthermore, our proposed texture generation technique can also be applied to texture editing while preserving the original identity. More experimental results are available at https://dong-huo.github.io/TexGen/