NPGA：神经参数高斯化身

NPGA: Neural Parametric Gaussian Avatars

May 29, 2024

作者: Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner

cs.AI

摘要

在进一步将虚拟组件融入日常生活的过程中，创建高保真度的数字化人头版本是一个重要的里程碑。构建这样的化身是一个具有挑战性的研究问题，因为对照片逼真度和实时渲染性能的需求很高。在这项工作中，我们提出了神经参数高斯化身（NPGA），这是一种数据驱动方法，可以从多视角视频录制中创建高保真度、可控制的化身。我们的方法基于3D高斯飞溅，因为它具有高效的渲染能力，并且继承了点云的拓扑灵活性。与先前的工作相反，我们将化身的动态调节到神经参数头部模型（NPHM）的丰富表情空间上，而不是基于网格的3DMM。为此，我们将底层NPHM的反向变形场提炼为与光栅化渲染兼容的正向变形。所有其余的细节，如表情相关细节，都是从多视角视频中学习的。为了增加我们的化身的表现能力，我们使用每个基元潜在特征来增强规范高斯点云，这些特征控制其动态行为。为了规范这种增强的动态表现力，我们在潜在特征和预测动态上提出了拉普拉斯项。我们在公开的NeRSemble数据集上评估了我们的方法，结果表明NPGA在自我再现任务中比先前最先进的化身表现提高了2.6 PSNR。此外，我们展示了从现实世界单眼视频中准确的动画能力。

English

The creation of high-fidelity, digital versions of human heads is an important stepping stone in the process of further integrating virtual components into our everyday lives. Constructing such avatars is a challenging research problem, due to a high demand for photo-realism and real-time rendering performance. In this work, we propose Neural Parametric Gaussian Avatars (NPGA), a data-driven approach to create high-fidelity, controllable avatars from multi-view video recordings. We build our method around 3D Gaussian Splatting for its highly efficient rendering and to inherit the topological flexibility of point clouds. In contrast to previous work, we condition our avatars' dynamics on the rich expression space of neural parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we distill the backward deformation field of our underlying NPHM into forward deformations which are compatible with rasterization-based rendering. All remaining fine-scale, expression-dependent details are learned from the multi-view videos. To increase the representational capacity of our avatars, we augment the canonical Gaussian point cloud using per-primitive latent features which govern its dynamic behavior. To regularize this increased dynamic expressivity, we propose Laplacian terms on the latent features and predicted dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating that NPGA significantly outperforms the previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate animation capabilities from real-world monocular videos.