NeRSemble：人頭多視角輻射場重建

摘要

我們專注於重建人類頭部的高保真輻射場，捕捉其隨時間變化的動畫，並從新的視角在任意時間步驟合成重新渲染。為此，我們提出了一種新的多視角捕獲設置，由16台校準的機器視覺相機組成，以7.1 MP的分辨率和每秒73幀記錄同步時間的影像。通過我們的設置，我們收集了一個新的數據集，包括超過4700個高分辨率、高幀率的序列，涵蓋了220多個人類頭部，並引入了一個新的人類頭部重建基準。記錄的序列涵蓋了廣泛的面部動態，包括頭部運動、自然表情、情感和口語。為了重建高保真度的人類頭部，我們提出了使用哈希集成的動態神經輻射場（NeRSemble）。我們通過結合變形場和一組3D多分辨率哈希編碼來表示場景動態。變形場可以精確建模簡單場景運動，而哈希編碼的集成有助於表示複雜的動態。因此，我們獲得了捕捉隨時間運動並促進重新渲染任意新視角的人類頭部輻射場表示。在一系列實驗中，我們探索了我們方法的設計選擇，並證明我們的方法在很大程度上優於最先進的動態輻射場方法。

English

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.

NeRSemble：人頭多視角輻射場重建

NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

摘要

Support