AvatarReX：实时表现力全身化身
AvatarReX: Real-time Expressive Full-body Avatars
May 8, 2023
作者: Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, Yebin Liu
cs.AI
摘要
我们提出了AvatarReX，这是一种从视频数据中学习基于NeRF的全身化身的新方法。学习到的化身不仅可以提供对身体、手部和面部的表现控制，还支持实时动画和渲染。为此，我们提出了一种组合式化身表示，其中身体、手部和面部分别建模，以便充分利用参数化网格模板的结构先验，同时不影响表示的灵活性。此外，我们对每个部分的几何和外观进行了解耦。通过这些技术设计，我们提出了一个专用的延迟渲染流水线，可以以实时帧率执行，合成高质量的自由视图图像。几何和外观的解耦还使我们能够设计一个两阶段训练策略，结合体积渲染和表面渲染进行网络训练。通过这种方式，可以应用基于补丁级别的监督，迫使网络学习基于几何估计的清晰外观细节。总体而言，我们的方法实现了具有实时渲染能力的表现丰富的全身化身的自动构建，并能够为新颖的身体动作和面部表情生成具有动态细节的逼真图像。
English
We present AvatarReX, a new method for learning NeRF-based full-body avatars
from video data. The learnt avatar not only provides expressive control of the
body, hands and the face together, but also supports real-time animation and
rendering. To this end, we propose a compositional avatar representation, where
the body, hands and the face are separately modeled in a way that the
structural prior from parametric mesh templates is properly utilized without
compromising representation flexibility. Furthermore, we disentangle the
geometry and appearance for each part. With these technical designs, we propose
a dedicated deferred rendering pipeline, which can be executed in real-time
framerate to synthesize high-quality free-view images. The disentanglement of
geometry and appearance also allows us to design a two-pass training strategy
that combines volume rendering and surface rendering for network training. In
this way, patch-level supervision can be applied to force the network to learn
sharp appearance details on the basis of geometry estimation. Overall, our
method enables automatic construction of expressive full-body avatars with
real-time rendering capability, and can generate photo-realistic images with
dynamic details for novel body motions and facial expressions.