TaoAvatar：基于3D高斯泼溅的实时逼真全身对话虚拟角色，面向增强现实应用

TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

March 21, 2025

作者: Jianchuan Chen, Jingchuan Hu, Gaige Wang, Zhonghua Jiang, Tiansong Zhou, Zhiwen Chen, Chengfei Lv

cs.AI

摘要

逼真的3D全身对话式虚拟人在增强现实（AR）领域展现出巨大潜力，其应用范围涵盖电商直播至全息通信。尽管3D高斯溅射（3DGS）技术在创建栩栩如生的虚拟人方面取得了进展，现有方法在全身对话任务中对面部表情和身体动作的精细控制上仍面临挑战。此外，这些方法往往缺乏足够的细节，且无法在移动设备上实时运行。我们提出了TaoAvatar，一个基于3DGS的高保真、轻量级全身对话式虚拟人，能够响应多种信号驱动。我们的方法首先构建了一个个性化的着装人体参数化模板，将高斯分布绑定以呈现外观。随后，我们预训练了一个基于StyleUnet的网络，以处理复杂的姿态依赖非刚性变形，该网络虽能捕捉高频外观细节，但对移动设备而言计算资源消耗过大。为解决这一问题，我们采用蒸馏技术将非刚性变形“烘焙”进一个轻量级的基于MLP的网络中，并开发了混合形状以补偿细节损失。大量实验表明，TaoAvatar在保持90 FPS于如Apple Vision Pro等高清晰度立体设备上的同时，实现了实时跨设备运行，并达到了业界领先的渲染质量。

English

Realistic 3D full-body talking avatars hold great potential in AR, with applications ranging from e-commerce live streaming to holographic communication. Despite advances in 3D Gaussian Splatting (3DGS) for lifelike avatar creation, existing methods struggle with fine-grained control of facial expressions and body movements in full-body talking tasks. Additionally, they often lack sufficient details and cannot run in real-time on mobile devices. We present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking avatar driven by various signals. Our approach starts by creating a personalized clothed human parametric template that binds Gaussians to represent appearances. We then pre-train a StyleUnet-based network to handle complex pose-dependent non-rigid deformation, which can capture high-frequency appearance details but is too resource-intensive for mobile devices. To overcome this, we "bake" the non-rigid deformations into a lightweight MLP-based network using a distillation technique and develop blend shapes to compensate for details. Extensive experiments show that TaoAvatar achieves state-of-the-art rendering quality while running in real-time across various devices, maintaining 90 FPS on high-definition stereo devices such as the Apple Vision Pro.