HeadSculpt：使用文本製作3D頭像

HeadSculpt: Crafting 3D Head Avatars with Text

June 5, 2023

作者: Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Kwan-Yee K. Wong

cs.AI

摘要

最近，受惠於大視覺語言和圖像擴散模型的普及，基於文本引導的3D生成方法在製作高質量紋理和幾何方面取得了顯著進展。然而，現有方法在兩個方面仍然難以創建高保真度的3D頭像：(1) 它們主要依賴預先訓練的文本到圖像擴散模型，卻缺乏必要的3D意識和頭部先驗知識。這使得生成的頭像容易出現不一致性和幾何扭曲。(2) 它們在細粒度編輯方面表現不佳。這主要是由於從預先訓練的2D圖像擴散模型繼承的限制，當涉及到3D頭像時這些限制變得更加明顯。在這項工作中，我們通過引入一個名為HeadSculpt的多功能從粗到細的流程，來應對這些挑戰，用於從文本提示中製作（即生成和編輯）3D頭像。具體來說，我們首先通過利用基於地標的控制和表示頭部背面外觀的學習文本嵌入，為擴散模型配備3D意識，從而實現3D一致的頭像生成。我們進一步提出一種新的身份感知編輯分數提煉策略，通過高分辨率的可微渲染技術來優化具有紋理的網格。這使得在遵循編輯指示的同時保持身份特徵。我們通過全面的實驗和與現有方法的比較展示了HeadSculpt卓越的保真度和編輯能力。

English

Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.