URHand：通用可重照手

URHand: Universal Relightable Hands

January 10, 2024

作者: Zhaoxi Chen, Gyeongsik Moon, Kaiwen Guo, Chen Cao, Stanislav Pidhorskyi, Tomas Simon, Rohan Joshi, Yuan Dong, Yichen Xu, Bernardo Pires, He Wen, Lucas Evans, Bo Peng, Julia Buffalini, Autumn Trimble, Kevyn McPhail, Melissa Schoeller, Shoou-I Yu, Javier Romero, Michael Zollhöfer, Yaser Sheikh, Ziwei Liu, Shunsuke Saito

cs.AI

摘要

現有的逼真可調光手部模型需要在不同視角、姿勢和光線下進行廣泛的特定身份觀察，並面臨著泛化到自然光線和新身份的挑戰。為彌合這一差距，我們提出了URHand，這是第一個可以泛化到不同視角、姿勢、光線和身份的通用可調光手部模型。我們的模型允許使用通過手機拍攝的圖像進行少量快速個性化，並準備好在新光線下進行逼真渲染。為了簡化個性化過程並保留逼真感，我們基於從具有數百個身份的光線舞台中拍攝的多視圖手部圖像進行神經照明的強大通用可調光先驗。關鍵挑戰在於在保持個性化忠實度和細節清晰度的同時擴展跨身份訓練，而不會影響在自然光線下的泛化。為此，我們提出了一個空間變化的線性照明模型作為神經渲染器，該模型以受物理啟發的著色作為輸入特徵。通過去除非線性激活和偏差，我們特別設計的照明模型明確保持了光線傳輸的線性性。這使得可以從光線舞台數據進行單階段訓練，同時在不同身份之間泛化到實時渲染下的任意連續光線。此外，我們介紹了基於物理的模型和我們的神經照明模型的聯合學習，進一步提高了忠實度和泛化性。大量實驗表明，我們的方法在質量和泛化能力方面均優於現有方法。我們還展示了如何從對未見身份進行簡短手機掃描快速個性化URHand。

English

Existing photorealistic relightable hand models require extensive identity-specific observations in different views, poses, and illuminations, and face challenges in generalizing to natural illuminations and novel identities. To bridge this gap, we present URHand, the first universal relightable hand model that generalizes across viewpoints, poses, illuminations, and identities. Our model allows few-shot personalization using images captured with a mobile phone, and is ready to be photorealistically rendered under novel illuminations. To simplify the personalization process while retaining photorealism, we build a powerful universal relightable prior based on neural relighting from multi-view images of hands captured in a light stage with hundreds of identities. The key challenge is scaling the cross-identity training while maintaining personalized fidelity and sharp details without compromising generalization under natural illuminations. To this end, we propose a spatially varying linear lighting model as the neural renderer that takes physics-inspired shading as input feature. By removing non-linear activations and bias, our specifically designed lighting model explicitly keeps the linearity of light transport. This enables single-stage training from light-stage data while generalizing to real-time rendering under arbitrary continuous illuminations across diverse identities. In addition, we introduce the joint learning of a physically based model and our neural relighting model, which further improves fidelity and generalization. Extensive experiments show that our approach achieves superior performance over existing methods in terms of both quality and generalizability. We also demonstrate quick personalization of URHand from a short phone scan of an unseen identity.