TalkinNeRF：用于全身说话人类的可动画神经场

TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

September 25, 2024

作者: Aggelina Chatziagapi, Bindita Chaudhuri, Amit Kumar, Rakesh Ranjan, Dimitris Samaras, Nikolaos Sarafianos

cs.AI

摘要

我们介绍了一个新颖的框架，从单目视频中学习全身说话人的动态神经辐射场（NeRF）。先前的研究仅表示身体姿势或面部。然而，人类通过全身进行交流，结合身体姿势、手势以及面部表情。在这项工作中，我们提出了TalkinNeRF，一个基于统一NeRF的网络，表示整体的4D人体运动。给定一个主体的单目视频，我们学习相应的身体、面部和手部模块，将它们组合在一起生成最终结果。为了捕捉复杂的手指关节运动，我们学习了额外的手部变形场。我们的多身份表示使得能够同时训练多个主体，并在完全看不见的姿势下进行稳健的动画。它还可以推广到新的身份，仅需短视频作为输入。我们展示了在为全身说话人进行动画时的最先进性能，具有精细的手部关节运动和面部表情。

English

We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions.