UltrAvatar：一种具有真实感的可动画3D头像扩散模型，带有真实性引导纹理。

UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures

January 20, 2024

作者: Mingyuan Zhou, Rakib Hyder, Ziwei Xuan, Guojun Qi

cs.AI

摘要

最近在3D头像生成方面取得了重要进展，引起了广泛关注。这些突破旨在生成更逼真、可动画化的头像，缩小虚拟和现实世界体验之间的差距。大多数现有作品采用得分蒸馏采样（SDS）损失，结合可微分渲染器和文本条件，指导扩散模型生成3D头像。然而，SDS通常会生成过度平滑的结果，面部细节较少，因此与祖先采样相比缺乏多样性。另一方面，其他作品从单个图像生成3D头像，面临不必要的光照效果、透视视图和图像质量较低等挑战，这使得它们难以可靠地重建具有对齐完整纹理的3D面部网格。在本文中，我们提出了一种名为UltrAvatar的新型3D头像生成方法，具有增强的几何保真度和优质的基于物理的渲染（PBR）纹理质量，且不受不必要的光照影响。为此，所提出的方法提出了一个扩散颜色提取模型和一个真实性引导纹理扩散模型。前者消除了不必要的光照效果，揭示真实的扩散颜色，使生成的头像能够在各种光照条件下渲染。后者遵循两个基于梯度的指导，用于生成PBR纹理，以更好地呈现多样的面部特征和细节，更好地与3D网格几何对齐。我们展示了所提出方法的有效性和鲁棒性，在实验中大幅优于现有方法。

English

Recent advances in 3D avatar generation have gained significant attentions. These breakthroughs aim to produce more realistic animatable avatars, narrowing the gap between virtual and real-world experiences. Most of existing works employ Score Distillation Sampling (SDS) loss, combined with a differentiable renderer and text condition, to guide a diffusion model in generating 3D avatars. However, SDS often generates oversmoothed results with few facial details, thereby lacking the diversity compared with ancestral sampling. On the other hand, other works generate 3D avatar from a single image, where the challenges of unwanted lighting effects, perspective views, and inferior image quality make them difficult to reliably reconstruct the 3D face meshes with the aligned complete textures. In this paper, we propose a novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR) textures without unwanted lighting. To this end, the proposed approach presents a diffuse color extraction model and an authenticity guided texture diffusion model. The former removes the unwanted lighting effects to reveal true diffuse colors so that the generated avatars can be rendered under various lighting conditions. The latter follows two gradient-based guidances for generating PBR textures to render diverse face-identity features and details better aligning with 3D mesh geometry. We demonstrate the effectiveness and robustness of the proposed method, outperforming the state-of-the-art methods by a large margin in the experiments.