可重新照明的全身高斯編碼頭像

Relightable Full-Body Gaussian Codec Avatars

January 24, 2025

作者: Shaofei Wang, Tomas Simon, Igor Santesteban, Timur Bagautdinov, Junxuan Li, Vasu Agrawal, Fabian Prada, Shoou-I Yu, Pace Nalbone, Matt Gramlich, Roman Lubachersky, Chenglei Wu, Javier Romero, Jason Saragih, Michael Zollhoefer, Andreas Geiger, Siyu Tang, Shunsuke Saito

cs.AI

摘要

我們提出了可重新照明的全身高斯編碼化身，這是一種新方法，用於建模包括臉部和手部在內的具有精細細節的可重新照明全身化身。重新照明全身化身的獨特挑戰在於由身體關節運動引起的大變形，以及對外觀造成的影響。身體姿勢的變化可以顯著改變身體表面相對於光源的方向，導致由於局部光傳輸函數的變化而導致的局部外觀變化，以及由於身體部位之間的遮蔽而導致的非局部變化。為了應對這一挑戰，我們將光傳輸分解為局部和非局部效應。局部外觀變化使用可學習的區域諧波來建模漫射輻射傳輸。與球諧波不同，區域諧波在關節運動下旋轉效率非常高。這使我們能夠在局部坐標系統中學習漫射輻射傳輸，從而將局部輻射傳輸與身體的關節運動分離。為了考慮非局部外觀變化，我們引入了一個陰影網絡，根據基本網格上預先計算的入射輻照度來預測陰影。這有助於學習身體部位之間的非局部陰影。最後，我們使用延遲着色方法來建模鏡面輻射傳輸，更好地捕捉反射和高光，如眼睛閃爍。我們展示了我們的方法成功地建模了可重新照明的全身化身所需的局部和非局部光傳輸，具有在新的照明條件和未見姿勢下優越的泛化能力。

English

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for modeling relightable full-body avatars with fine-grained details including face and hands. The unique challenge for relighting full-body avatars lies in the large deformations caused by body articulation and the resulting impact on appearance caused by light transport. Changes in body pose can dramatically change the orientation of body surfaces with respect to lights, resulting in both local appearance changes due to changes in local light transport functions, as well as non-local changes due to occlusion between body parts. To address this, we decompose the light transport into local and non-local effects. Local appearance changes are modeled using learnable zonal harmonics for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are highly efficient to rotate under articulation. This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body. To account for non-local appearance changes, we introduce a shadow network that predicts shadows given precomputed incoming irradiance on a base mesh. This facilitates the learning of non-local shadowing between the body parts. Finally, we use a deferred shading approach to model specular radiance transfer and better capture reflections and highlights such as eye glints. We demonstrate that our approach successfully models both the local and non-local light transport required for relightable full-body avatars, with a superior generalization ability under novel illumination conditions and unseen poses.