可控的動態外觀用於神經網絡3D肖像。
Controllable Dynamic Appearance for Neural 3D Portraits
September 20, 2023
作者: ShahRukh Athar, Zhixin Shu, Zexiang Xu, Fuji Luan, Sai Bi, Kalyan Sunkavalli, Dimitris Samaras
cs.AI
摘要
最近在神經輻射場（Neural Radiance Fields，NeRFs）方面的進展使得重建和重新製作動態肖像場景成為可能，並可控制頭部姿勢、面部表情和觀看方向。然而，訓練這些模型需要假設變形區域內的光度一致性，例如，隨著頭部姿勢和面部表情的變化，臉部必須均勻照亮。即使在工作室環境中，跨視頻幀的這種光度一致性很難保持，因此在重新製作過程中，創建的可重新製作的神經肖像容易出現瑕疵。在這項工作中，我們提出了CoDyNeRF，這是一個系統，可以在現實世界的拍攝條件下創建完全可控的3D肖像。CoDyNeRF通過在規範空間中的動態外觀模型來學習近似光線依賴效應，該模型受到預測表面法線以及面部表情和頭部姿勢變形的條件影響。表面法線的預測是通過作為人頭法線的粗略先驗的3DMM法線來引導的，由於頭部姿勢和面部表情變化引起的剛性和非剛性變形，直接預測法線是困難的。通過僅使用智能手機拍攝的被試者短視頻進行訓練，我們展示了我們的方法在具有明確頭部姿勢和表情控制以及逼真照明效果的肖像場景的自由視圖合成方面的有效性。項目頁面可在此處找到：http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html
English
Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
reconstruct and reanimate dynamic portrait scenes with control over head-pose,
facial expressions and viewing direction. However, training such models assumes
photometric consistency over the deformed region e.g. the face must be evenly
lit as it deforms with changing head-pose and facial expression. Such
photometric consistency across frames of a video is hard to maintain, even in
studio environments, thus making the created reanimatable neural portraits
prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
system that enables the creation of fully controllable 3D portraits in
real-world capture conditions. CoDyNeRF learns to approximate illumination
dependent effects via a dynamic appearance model in the canonical space that is
conditioned on predicted surface normals and the facial expressions and
head-pose deformations. The surface normals prediction is guided using 3DMM
normals that act as a coarse prior for the normals of the human head, where
direct prediction of normals is hard due to rigid and non-rigid deformations
induced by head-pose and facial expression changes. Using only a
smartphone-captured short video of a subject for training, we demonstrate the
effectiveness of our method on free view synthesis of a portrait scene with
explicit head pose and expression controls, and realistic lighting effects. The
project page can be found here:
http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html