TexFusion：使用文本引导的图像扩散模型合成3D纹理

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

October 20, 2023

作者: Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin

cs.AI

摘要

我们提出了TexFusion（纹理扩散），这是一种新方法，用于合成给定3D几何形状的纹理，利用大规模文本引导的图像扩散模型。与最近利用2D文本到图像扩散模型来提炼3D对象的作品不同，这些作品使用缓慢且脆弱的优化过程，TexFusion引入了一种新的3D一致生成技术，专门设计用于纹理合成，它采用不同2D渲染视图上的常规扩散模型采样。具体而言，我们利用潜在扩散模型，在3D对象的一组2D渲染上应用扩散模型的去噪器，并将不同的去噪预测聚合在共享的潜在纹理地图上。通过在潜在纹理的2D渲染的解码上优化中间神经颜色场，生成最终的RGB纹理。我们对TexFusion进行了彻底验证，并展示我们可以高效生成多样化、高质量和全局一致的纹理。我们仅使用图像扩散模型实现了最先进的文本引导纹理合成性能，同时避免了以往基于提炼的方法的缺陷。文本调节提供了详细控制，我们也不依赖任何真实的3D纹理进行训练。这使得我们的方法多才多艺，适用于广泛的几何和纹理类型。我们希望TexFusion将推动基于人工智能的3D资产纹理化在虚拟现实、游戏设计、模拟等应用中的发展。

English

We present TexFusion (Texture Diffusion), a new method to synthesize textures for given 3D geometries, using large-scale text-guided image diffusion models. In contrast to recent works that leverage 2D text-to-image diffusion models to distill 3D objects using a slow and fragile optimization process, TexFusion introduces a new 3D-consistent generation technique specifically designed for texture synthesis that employs regular diffusion model sampling on different 2D rendered views. Specifically, we leverage latent diffusion models, apply the diffusion model's denoiser on a set of 2D renders of the 3D object, and aggregate the different denoising predictions on a shared latent texture map. Final output RGB textures are produced by optimizing an intermediate neural color field on the decodings of 2D renders of the latent texture. We thoroughly validate TexFusion and show that we can efficiently generate diverse, high quality and globally coherent textures. We achieve state-of-the-art text-guided texture synthesis performance using only image diffusion models, while avoiding the pitfalls of previous distillation-based methods. The text-conditioning offers detailed control and we also do not rely on any ground truth 3D textures for training. This makes our method versatile and applicable to a broad range of geometry and texture types. We hope that TexFusion will advance AI-based texturing of 3D assets for applications in virtual reality, game design, simulation, and more.