可重新照明的高斯編碼頭像

摘要

重新照明的忠實度受到幾何和外觀表示的限制。在幾何方面，網格和體積方法都難以建模複雜的結構，如三維頭髮幾何。對於外觀，現有的重新照明模型在忠實度上受限，並且通常在高分辨率連續環境中渲染實時時速過慢。在這項工作中，我們提出了可重新照明的高忠實度頭像的高斯編碼化身方法，這些頭像可以被動畫化以生成新的表情。我們基於三維高斯模型的幾何模型可以捕捉三維一致的亞毫米細節，例如動態面部序列上的頭髮細絲和毛孔。為了以統一的方式支持人類頭部的各種材料，如眼睛、皮膚和頭髮，我們提出了一種基於可學習輻射傳輸的新型可重新照明外觀模型。結合用於漫射成分的全局照明感知球面調和，我們使用球形高斯實現了具有空間所有頻率反射的實時重新照明。這種外觀模型可以在點光源和連續照明下高效地重新照明。我們通過引入可重新照明的明確眼睛模型進一步提高了眼睛反射的忠實度並實現了明確的凝視控制。我們的方法在不影響實時性能的情況下勝過現有方法。我們還展示了在一個連接的消費者VR頭戴設備上對化身進行實時重新照明，展示了我們化身的效率和忠實度。

English

The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intricate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution continuous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel relightable appearance model based on learnable radiance transfer. Together with global illumination-aware spherical harmonics for the diffuse components, we achieve real-time relighting with spatially all-frequency reflections using spherical Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches without compromising real-time performance. We also demonstrate real-time relighting of avatars on a tethered consumer VR headset, showcasing the efficiency and fidelity of our avatars.