SEEAvatar：基于约束几何和外观的照片逼真文本到3D头像生成

SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance

December 13, 2023

作者: Yuanyou Xu, Zongxin Yang, Yi Yang

cs.AI

摘要

借助大规模文本到图像生成模型的支持，文本到3D头像生成已取得令人期待的进展。然而，大多数方法未能产生逼真的结果，受到不精确的几何形状和低质量外观的限制。为了更实用的头像生成，我们提出了SEEAvatar，一种从文本生成逼真3D头像的方法，其中包含用于解耦几何形状和外观的SElf-Evolving约束。对于几何形状，我们建议通过将优化后的头像与模板头像在良好的全局形状上进行约束来生成头像。模板头像以人类先验初始化，并可以定期由优化后的头像更新为演变模板，从而实现更灵活的形状生成。此外，几何形状还受到脸部和手部等局部的静态人类先验的约束，以保持精细的结构。对于外观生成，我们使用扩散模型结合提示工程来引导基于物理的渲染管线生成逼真的纹理。光亮度约束应用于反照率纹理，以抑制不正确的光照效果。实验证明，我们的方法在全局和局部几何形状以及外观质量方面均大幅优于先前的方法。由于我们的方法可以生成高质量的网格和纹理，这些资源可以直接应用于经典图形管线中，在任何光照条件下进行逼真渲染。项目页面网址：https://seeavatar3d.github.io。

English

Powered by large-scale text-to-image generation models, text-to-3D avatar generation has made promising progress. However, most methods fail to produce photorealistic results, limited by imprecise geometry and low-quality appearance. Towards more practical avatar generation, we present SEEAvatar, a method for generating photorealistic 3D avatars from text with SElf-Evolving constraints for decoupled geometry and appearance. For geometry, we propose to constrain the optimized avatar in a decent global shape with a template avatar. The template avatar is initialized with human prior and can be updated by the optimized avatar periodically as an evolving template, which enables more flexible shape generation. Besides, the geometry is also constrained by the static human prior in local parts like face and hands to maintain the delicate structures. For appearance generation, we use diffusion model enhanced by prompt engineering to guide a physically based rendering pipeline to generate realistic textures. The lightness constraint is applied on the albedo texture to suppress incorrect lighting effect. Experiments show that our method outperforms previous methods on both global and local geometry and appearance quality by a large margin. Since our method can produce high-quality meshes and textures, such assets can be directly applied in classic graphics pipeline for realistic rendering under any lighting condition. Project page at: https://seeavatar3d.github.io.