用于可重光和可交互面部渲染的即时面部高斯翻译器

Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering

September 11, 2024

作者: Dafei Qin, Hongyang Lin, Qixuan Zhang, Kaichun Qiao, Longwen Zhang, Zijun Zhao, Jun Saito, Jingyi Yu, Lan Xu, Taku Komura

cs.AI

摘要

我们提出了GauFace，一种新颖的高斯飞溅表示法，专为高效动画和渲染基于物理的面部资产而设计。利用强大的几何先验和受限优化，GauFace确保了整洁和结构化的高斯表示，提供了高保真度和实时面部交互，可在骁龙8 Gen 2移动平台上以30fps@1440p的速度运行。 然后，我们介绍了TransGS，一种扩散变换器，可以即时将基于物理的面部资产转换为相应的GauFace表示。具体来说，我们采用基于补丁的流水线来有效处理大量的高斯函数。我们还引入了一种新颖的像素对齐采样方案，结合UV位置编码，以确保由我们的TransGS生成的GauFace资产的吞吐量和渲染质量。一经训练，TransGS可以即时将带有光照条件的面部资产转换为GauFace表示。通过丰富的调节模式，它还能够实现类似传统CG流水线的编辑和动画功能。 我们进行了广泛的评估和用户研究，与传统的离线和在线渲染器以及最近的神经渲染方法进行了比较，结果显示我们的方法在面部资产渲染方面表现出卓越的性能。我们还展示了使用我们的TransGS方法和GauFace表示的面部资产在各种平台上的多样沉浸式应用，包括PC、手机甚至VR头显。

English

We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.