HeadSculpt：使用文本创建3D头像

HeadSculpt: Crafting 3D Head Avatars with Text

June 5, 2023

作者: Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Kwan-Yee K. Wong

cs.AI

摘要

最近，文本引导的3D生成方法在生成高质量纹理和几何方面取得了显著进展，充分利用了大规模视觉-语言和图像扩散模型的普及。然而，现有方法仍然在两个方面难以创建高保真度的3D头像：(1) 它们主要依赖预训练的文本到图像扩散模型，缺乏必要的3D意识和头部先验知识。这使它们在生成头像时容易出现不一致性和几何失真。(2) 它们在细粒度编辑方面表现不佳。这主要是由于从预训练的2D图像扩散模型继承的限制，当涉及到3D头像时，这些限制变得更加明显。在这项工作中，我们通过引入一种名为HeadSculpt的多功能粗到细的流程来解决这些挑战，用于从文本提示中塑造（即生成和编辑）3D头像。具体而言，我们首先通过利用基于地标的控制和表示头部背面外观的学习文本嵌入，为扩散模型配备3D意识，从而实现一致的3D头像生成。我们进一步提出了一种新颖的身份感知编辑评分蒸馏策略，通过优化具有高分辨率可微渲染技术的纹理网格，实现身份保留并遵循编辑指令。我们通过全面的实验和与现有方法的比较展示了HeadSculpt卓越的保真度和编辑能力。

English

Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.