实现高保真可重光化头像的实用捕捉路径

摘要

本文提出了一种新颖的框架，称为无追踪可重光化头像（TRAvatar），用于捕捉和重建高保真度的3D头像。与先前的方法相比，TRAvatar在更实用和高效的环境中运行。具体而言，TRAvatar使用在光线舞台下捕获的动态图像序列进行训练，这些图像序列在不同光照条件下进行，从而实现头像在各种场景中的逼真重光和实时动画。此外，TRAvatar允许无追踪头像捕获，并消除了在不同光照条件下准确表面跟踪的需求。我们的贡献有两个方面：首先，我们提出了一种新颖的网络架构，明确建立并确保光照的线性特性。在简单的光组捕获训练下，TRAvatar可以通过单次前向传递预测实时外观，实现在任意环境贴图照明下的高质量重光效果。其次，我们基于图像序列从零开始联合优化面部几何和可重光外观，其中追踪是隐式学习的。这种无追踪方法增强了在不同光照条件下建立帧间时间对应关系的稳健性。大量定性和定量实验表明，我们的框架在逼真头像动画和重光方面实现了卓越性能。

English

In this paper, we propose a novel framework, Tracking-free Relightable Avatar (TRAvatar), for capturing and reconstructing high-fidelity 3D avatars. Compared to previous methods, TRAvatar works in a more practical and efficient setting. Specifically, TRAvatar is trained with dynamic image sequences captured in a Light Stage under varying lighting conditions, enabling realistic relighting and real-time animation for avatars in diverse scenes. Additionally, TRAvatar allows for tracking-free avatar capture and obviates the need for accurate surface tracking under varying illumination conditions. Our contributions are two-fold: First, we propose a novel network architecture that explicitly builds on and ensures the satisfaction of the linear nature of lighting. Trained on simple group light captures, TRAvatar can predict the appearance in real-time with a single forward pass, achieving high-quality relighting effects under illuminations of arbitrary environment maps. Second, we jointly optimize the facial geometry and relightable appearance from scratch based on image sequences, where the tracking is implicitly learned. This tracking-free approach brings robustness for establishing temporal correspondences between frames under different lighting conditions. Extensive qualitative and quantitative experiments demonstrate that our framework achieves superior performance for photorealistic avatar animation and relighting.

实现高保真可重光化头像的实用捕捉路径

Towards Practical Capture of High-Fidelity Relightable Avatars

摘要

Support