HumanSplat：具有结构先验知识的通用单图像人类高斯飘带

HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

June 18, 2024

作者: Panwang Pan, Zhuo Su, Chenguo Lin, Zhen Fan, Yongjie Zhang, Zeming Li, Tingting Shen, Yadong Mu, Yebin Liu

cs.AI

摘要

尽管最近高保真人体重建技术取得了进展，但对密集捕捉图像或耗时的每个实例优化的要求显著阻碍了它们在更广泛场景中的应用。为了解决这些问题，我们提出了HumanSplat，它以通用的方式预测任何人的三维高斯Splatting属性，仅从单个输入图像中进行预测。具体而言，HumanSplat 包括一个二维多视图扩散模型和一个具有人体结构先验的潜在重建变换器，巧妙地在统一框架内整合几何先验和语义特征。进一步设计了一个包含人体语义信息的分层损失，以实现高保真纹理建模并更好地约束估计的多视图。对标准基准和野外图像的全面实验表明，HumanSplat 在实现逼真的新视角合成方面超越了现有的最先进方法。

English

Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present HumanSplat which predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner. In particular, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is further designed to achieve high-fidelity texture modeling and better constrain the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis.