神经三维肖像的可控动态外观

摘要

最近在神经辐射场（NeRFs）方面取得的进展使得重建和重新激活动态肖像场景成为可能，可以控制头部姿势、面部表情和观看方向。然而，训练这样的模型假定在变形区域上存在光度一致性，例如，面部在随着头部姿势和面部表情变化而变形时必须均匀照亮。即使在工作室环境中，跨视频帧的这种光度一致性也很难维持，因此在重新激活过程中创造的可重新激活的神经肖像容易出现瑕疵。在这项工作中，我们提出了CoDyNeRF，这是一个系统，可以在真实世界的拍摄条件下创建完全可控的3D肖像。CoDyNeRF通过在规范空间中的动态外观模型来学习近似光照相关效果，该模型是根据预测的表面法线、面部表情和头部姿势变形进行条件化的。表面法线的预测是通过作为人头表面法线的粗略先验的3DMM法线来引导的，由于头部姿势和面部表情变化引起的刚性和非刚性变形，直接预测法线是困难的。仅使用智能手机捕获的主体的短视频进行训练，我们展示了我们的方法在具有明确头部姿势和表情控制以及逼真光照效果的肖像场景的自由视图合成方面的有效性。项目页面链接：http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html

English

Recent advances in Neural Radiance Fields (NeRFs) have made it possible to reconstruct and reanimate dynamic portrait scenes with control over head-pose, facial expressions and viewing direction. However, training such models assumes photometric consistency over the deformed region e.g. the face must be evenly lit as it deforms with changing head-pose and facial expression. Such photometric consistency across frames of a video is hard to maintain, even in studio environments, thus making the created reanimatable neural portraits prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a system that enables the creation of fully controllable 3D portraits in real-world capture conditions. CoDyNeRF learns to approximate illumination dependent effects via a dynamic appearance model in the canonical space that is conditioned on predicted surface normals and the facial expressions and head-pose deformations. The surface normals prediction is guided using 3DMM normals that act as a coarse prior for the normals of the human head, where direct prediction of normals is hard due to rigid and non-rigid deformations induced by head-pose and facial expression changes. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls, and realistic lighting effects. The project page can be found here: http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html

神经三维肖像的可控动态外观

Controllable Dynamic Appearance for Neural 3D Portraits

摘要

Support