NeRSemble：人头多视角辐射场重建

摘要

我们专注于重建人类头部的高保真辐射场，捕捉其随时间的动画，并从任意时间步长的新视角合成重新渲染。为此，我们提出了一个新的多视角捕捉设置，由16台经校准的机器视觉摄像头组成，以7.1 MP分辨率和每秒73帧记录同步图像。通过我们的设置，我们收集了一个包含超过4700个高分辨率、高帧率序列的新数据集，涵盖了220多个人类头部，从中我们引入了一个新的人类头部重建基准。记录的序列涵盖了广泛的面部动态，包括头部运动、自然表情、情绪和口语。为了重建高保真度的人类头部，我们提出了使用哈希集成的动态神经辐射场（NeRSemble）。我们通过结合变形场和一组3D多分辨率哈希编码来表示场景动态。变形场允许精确建模简单场景运动，而哈希编码的集成有助于表示复杂动态。因此，我们获得了捕捉随时间运动并促进重新渲染任意新视角的人类头部辐射场表示。在一系列实验中，我们探索了我们方法的设计选择，并证明我们的方法在很大程度上优于最先进的动态辐射场方法。

English

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.

NeRSemble：人头多视角辐射场重建

NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

摘要

Support