TexFusion：使用文本引導的圖像擴散模型合成3D紋理

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

October 20, 2023

作者: Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin

cs.AI

摘要

我們提出了TexFusion（紋理擴散），這是一種新方法，用於合成給定3D幾何形狀的紋理，採用大規模文本引導的圖像擴散模型。與最近利用2D文本到圖像擴散模型來提煉3D物體的作品相比，這些作品使用緩慢且脆弱的優化過程，TexFusion引入了一種新的3D一致生成技術，專門設計用於紋理合成，採用不同2D渲染視圖上的常規擴散模型抽樣。具體來說，我們利用潛在擴散模型，在3D物體的一組2D渲染上應用擴散模型的去噪器，並在共享的潛在紋理地圖上聚合不同的去噪預測。最終輸出的RGB紋理是通過在潛在紋理的2D渲染的解碼上優化中間神經顏色場來生成的。我們對TexFusion進行了全面驗證，並展示我們可以高效生成多樣、高質量和全局一致的紋理。我們僅使用圖像擴散模型實現了最先進的文本引導紋理合成性能，同時避免了先前基於提煉的方法的缺點。文本條件提供了詳細的控制，我們也不依賴任何真實的3D紋理進行訓練。這使得我們的方法多才多藝，適用於廣泛的幾何形狀和紋理類型。我們希望TexFusion將推動基於人工智慧的3D資產紋理應用，包括虛擬現實、遊戲設計、模擬等領域。

English

We present TexFusion (Texture Diffusion), a new method to synthesize textures for given 3D geometries, using large-scale text-guided image diffusion models. In contrast to recent works that leverage 2D text-to-image diffusion models to distill 3D objects using a slow and fragile optimization process, TexFusion introduces a new 3D-consistent generation technique specifically designed for texture synthesis that employs regular diffusion model sampling on different 2D rendered views. Specifically, we leverage latent diffusion models, apply the diffusion model's denoiser on a set of 2D renders of the 3D object, and aggregate the different denoising predictions on a shared latent texture map. Final output RGB textures are produced by optimizing an intermediate neural color field on the decodings of 2D renders of the latent texture. We thoroughly validate TexFusion and show that we can efficiently generate diverse, high quality and globally coherent textures. We achieve state-of-the-art text-guided texture synthesis performance using only image diffusion models, while avoiding the pitfalls of previous distillation-based methods. The text-conditioning offers detailed control and we also do not rely on any ground truth 3D textures for training. This makes our method versatile and applicable to a broad range of geometry and texture types. We hope that TexFusion will advance AI-based texturing of 3D assets for applications in virtual reality, game design, simulation, and more.