TexGen：使用多视角采样和重采样进行文本引导的3D纹理生成

TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling

August 2, 2024

作者: Dong Huo, Zixin Guo, Xinxin Zuo, Zhihao Shi, Juwei Lu, Peng Dai, Songcen Xu, Li Cheng, Yee-Hong Yang

cs.AI

摘要

针对给定的3D网格，我们旨在合成与任意文本描述相对应的3D纹理。当前用于从采样视图生成和组装纹理的方法通常会导致明显的接缝或过度平滑。为了解决这些问题，我们提出了TexGen，这是一个新颖的多视角采样和重采样框架，用于纹理生成，利用了预训练的文本到图像扩散模型。为了实现视角一致的采样，首先我们维护一个在RGB空间中参数化的纹理映射，该映射由去噪步骤参数化，并在每个扩散模型的采样步骤之后更新，逐渐减少视角差异。利用基于注意力的多视角采样策略来在视角之间传播外观信息。为了保留纹理细节，我们开发了一种噪声重采样技术，有助于噪声估计，生成用于后续去噪步骤的输入，根据文本提示和当前纹理映射的指导。通过大量的定性和定量评估，我们展示了我们提出的方法在各种3D对象的纹理质量方面表现出色，具有高度的视角一致性和丰富的外观细节，优于当前最先进的方法。此外，我们提出的纹理生成技术还可以应用于纹理编辑，同时保留原始身份。更多实验结果请访问https://dong-huo.github.io/TexGen/。

English

Given a 3D mesh, we aim to synthesize 3D textures that correspond to arbitrary textual descriptions. Current methods for generating and assembling textures from sampled views often result in prominent seams or excessive smoothing. To tackle these issues, we present TexGen, a novel multi-view sampling and resampling framework for texture generation leveraging a pre-trained text-to-image diffusion model. For view consistent sampling, first of all we maintain a texture map in RGB space that is parameterized by the denoising step and updated after each sampling step of the diffusion model to progressively reduce the view discrepancy. An attention-guided multi-view sampling strategy is exploited to broadcast the appearance information across views. To preserve texture details, we develop a noise resampling technique that aids in the estimation of noise, generating inputs for subsequent denoising steps, as directed by the text prompt and current texture map. Through an extensive amount of qualitative and quantitative evaluations, we demonstrate that our proposed method produces significantly better texture quality for diverse 3D objects with a high degree of view consistency and rich appearance details, outperforming current state-of-the-art methods. Furthermore, our proposed texture generation technique can also be applied to texture editing while preserving the original identity. More experimental results are available at https://dong-huo.github.io/TexGen/