URHand：通用可重照手

URHand: Universal Relightable Hands

January 10, 2024

作者: Zhaoxi Chen, Gyeongsik Moon, Kaiwen Guo, Chen Cao, Stanislav Pidhorskyi, Tomas Simon, Rohan Joshi, Yuan Dong, Yichen Xu, Bernardo Pires, He Wen, Lucas Evans, Bo Peng, Julia Buffalini, Autumn Trimble, Kevyn McPhail, Melissa Schoeller, Shoou-I Yu, Javier Romero, Michael Zollhöfer, Yaser Sheikh, Ziwei Liu, Shunsuke Saito

cs.AI

摘要

现有的逼真的可重光手部模型需要在不同视角、姿势和光照下进行广泛的特定身份观察，并面临着在自然光照和新身份上推广的挑战。为了弥合这一差距，我们提出了URHand，这是第一个能够跨视角、姿势、光照和身份进行泛化的通用可重光手部模型。我们的模型允许使用手机拍摄的图像进行少样本个性化，并能在新光照下逼真渲染。为了简化个性化过程并保持逼真性，我们基于来自灯光舞台上捕捉的数百个身份手部多视角图像的神经重光构建了强大的通用可重光先验。关键挑战在于在保持个性化保真度和锐利细节的同时扩展跨身份训练，而不损害在自然光照下的泛化能力。为此，我们提出了一个空间变化的线性光照模型作为神经渲染器，以物理启发的阴影作为输入特征。通过去除非线性激活和偏差，我们专门设计的光照模型明确保持了光传输的线性性。这使得可以从灯光舞台数据进行单阶段训练，同时在不同身份之间实现对任意连续光照的实时渲染泛化。此外，我们引入了基于物理的模型和我们的神经重光模型的联合学习，进一步提高了保真度和泛化性能。大量实验证明，我们的方法在质量和泛化能力方面均优于现有方法。我们还展示了如何通过对未知身份进行短时间手机扫描来快速个性化URHand。

English

Existing photorealistic relightable hand models require extensive identity-specific observations in different views, poses, and illuminations, and face challenges in generalizing to natural illuminations and novel identities. To bridge this gap, we present URHand, the first universal relightable hand model that generalizes across viewpoints, poses, illuminations, and identities. Our model allows few-shot personalization using images captured with a mobile phone, and is ready to be photorealistically rendered under novel illuminations. To simplify the personalization process while retaining photorealism, we build a powerful universal relightable prior based on neural relighting from multi-view images of hands captured in a light stage with hundreds of identities. The key challenge is scaling the cross-identity training while maintaining personalized fidelity and sharp details without compromising generalization under natural illuminations. To this end, we propose a spatially varying linear lighting model as the neural renderer that takes physics-inspired shading as input feature. By removing non-linear activations and bias, our specifically designed lighting model explicitly keeps the linearity of light transport. This enables single-stage training from light-stage data while generalizing to real-time rendering under arbitrary continuous illuminations across diverse identities. In addition, we introduce the joint learning of a physically based model and our neural relighting model, which further improves fidelity and generalization. Extensive experiments show that our approach achieves superior performance over existing methods in terms of both quality and generalizability. We also demonstrate quick personalization of URHand from a short phone scan of an unseen identity.