朝向實現高保真度可重新照明的虛擬人物的實用捕捉
Towards Practical Capture of High-Fidelity Relightable Avatars
September 8, 2023
作者: Haotian Yang, Mingwu Zheng, Wanquan Feng, Haibin Huang, Yu-Kun Lai, Pengfei Wan, Zhongyuan Wang, Chongyang Ma
cs.AI
摘要
本文提出了一個新穎的框架,稱為無追蹤可重燈化頭像(TRAvatar),用於捕捉和重建高保真度的3D頭像。相較於先前的方法,TRAvatar在更實用和高效的環境中運作。具體而言,TRAvatar是通過在不同照明條件下在光學舞台捕捉的動態圖像序列進行訓練,實現了多樣場景中頭像的逼真重燈和實時動畫。此外,TRAvatar允許無追蹤的頭像捕捉,並且無需在不同照明條件下進行準確的表面追蹤。我們的貢獻有兩個方面:首先,我們提出了一個新穎的網絡架構,明確建立在並確保照明的線性特性。在簡單的組光捕捉訓練下,TRAvatar可以通過單次前向傳播預測實時外觀,實現在任意環境地圖照明下的高質量重燈效果。其次,我們基於圖像序列從頭開始聯合優化面部幾何和可重燈外觀,其中追蹤是隱式學習的。這種無追蹤方法確保在不同照明條件下建立幀間時間對應的穩健性。大量定性和定量實驗證明,我們的框架在逼真頭像動畫和重燈方面實現了卓越性能。
English
In this paper, we propose a novel framework, Tracking-free Relightable Avatar
(TRAvatar), for capturing and reconstructing high-fidelity 3D avatars. Compared
to previous methods, TRAvatar works in a more practical and efficient setting.
Specifically, TRAvatar is trained with dynamic image sequences captured in a
Light Stage under varying lighting conditions, enabling realistic relighting
and real-time animation for avatars in diverse scenes. Additionally, TRAvatar
allows for tracking-free avatar capture and obviates the need for accurate
surface tracking under varying illumination conditions. Our contributions are
two-fold: First, we propose a novel network architecture that explicitly builds
on and ensures the satisfaction of the linear nature of lighting. Trained on
simple group light captures, TRAvatar can predict the appearance in real-time
with a single forward pass, achieving high-quality relighting effects under
illuminations of arbitrary environment maps. Second, we jointly optimize the
facial geometry and relightable appearance from scratch based on image
sequences, where the tracking is implicitly learned. This tracking-free
approach brings robustness for establishing temporal correspondences between
frames under different lighting conditions. Extensive qualitative and
quantitative experiments demonstrate that our framework achieves superior
performance for photorealistic avatar animation and relighting.