可重新点亮的高斯编解码头像

摘要

重新照明的保真度受几何和外观表示的限制。对于几何来说，无论是网格还是体积方法都难以建模像3D头发几何这样复杂的结构。对于外观来说，现有的重新照明模型在保真度上存在局限，并且通常渲染实时高分辨率连续环境时速度太慢。在这项工作中，我们提出了可重新照明的高保真度头像化编码化身，这是一种构建高保真度可重新照明头像的方法，可以进行动画处理以生成新颖的表情。我们基于3D高斯模型的几何模型可以捕捉动态面部序列上的毫米级细节，如头发细丝和毛孔等3D一致性细节。为了以统一方式支持人类头部的各种材料，如眼睛、皮肤和头发，我们提出了一种基于可学习辐射传输的新颖可重新照明外观模型。结合全局照明感知的球谐函数用于漫反射分量，我们利用球形高斯函数实现了具有空间所有频率反射的实时重新照明。这种外观模型可以在点光源和连续照明下高效地重新照明。我们进一步提高了眼睛反射的保真度，并通过引入可重新照明的显式眼睛模型实现了明确的凝视控制。我们的方法在不影响实时性能的情况下优于现有方法。我们还展示了在连接的消费者VR头显上对化身进行实时重新照明，展示了我们化身的高效性和保真度。

English

The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intricate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution continuous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel relightable appearance model based on learnable radiance transfer. Together with global illumination-aware spherical harmonics for the diffuse components, we achieve real-time relighting with spatially all-frequency reflections using spherical Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches without compromising real-time performance. We also demonstrate real-time relighting of avatars on a tethered consumer VR headset, showcasing the efficiency and fidelity of our avatars.