即時面部高斯翻譯器，用於可重新照明和互動的面部渲染。

Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering

September 11, 2024

作者: Dafei Qin, Hongyang Lin, Qixuan Zhang, Kaichun Qiao, Longwen Zhang, Zijun Zhao, Jun Saito, Jingyi Yu, Lan Xu, Taku Komura

cs.AI

摘要

我們提出了 GauFace，一種新穎的高斯點陣表示法，專為高效動畫和渲染基於物理的面部資產而設。通過利用強大的幾何先驗和受限優化，GauFace確保了整潔結構的高斯表示，實現了在Snapdragon 8 Gen 2移動平台上以30fps@1440p的實時面部互動的高保真度。 接著，我們介紹了TransGS，一種擴散變換器，可將基於物理的面部資產即時轉換為相應的GauFace表示。具體來說，我們採用基於補丁的流程來有效處理大量的高斯。我們還引入了一種新穎的像素對齊取樣方案，搭配UV位置編碼，以確保由我們的TransGS生成的GauFace資產的吞吐量和渲染質量。一旦訓練完成，TransGS可以立即將帶有照明條件的面部資產轉換為GauFace表示，憑藉豐富的條件模式，它還能實現類似傳統CG流程的編輯和動畫功能。 我們進行了廣泛的評估和用戶研究，與傳統的離線和在線渲染器以及最近的神經渲染方法進行了比較，這些顯示了我們的方法在面部資產渲染方面的優越性能。我們還展示了使用我們的TransGS方法和GauFace表示的面部資產在各種平台上（如PC、手機甚至VR頭戴設備）上的多樣沉浸式應用。

English

We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.