感知精準的3D說話頭像生成：新定義、語音網格表示與評估指標

Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics

March 26, 2025

作者: Lee Chae-Yeon, Oh Hyun-Bin, Han EunGi, Kim Sung-Bin, Suekyeong Nam, Tae-Hyun Oh

cs.AI

摘要

近期，語音驅動的三維說話頭像生成技術在唇形同步方面取得了顯著進展。然而，現有模型在捕捉不同語音特徵與相應唇部動作之間的感知對齊方面仍存在困難。在本研究中，我們提出三個標準——時間同步性、唇部可讀性和表現力——對於實現感知上精確的唇部動作至關重要。基於我們假設存在一個理想的表示空間來滿足這三個標準，我們引入了一種語音-網格同步表示，該表示捕捉了語音信號與三維面部網格之間的細微對應關係。我們發現，所學習的表示展現出理想的特性，並將其作為感知損失函數嵌入現有模型中，以更好地將唇部動作與給定語音對齊。此外，我們利用這一表示作為感知度量，並引入了另外兩個基於物理的唇形同步度量，以評估生成的三維說話頭像在多大程度上符合這三個標準。實驗表明，使用我們的感知損失函數訓練三維說話頭像生成模型，顯著提升了感知精確唇形同步的所有三個方面。代碼和數據集可在https://perceptual-3d-talking-head.github.io/獲取。

English

Recent advancements in speech-driven 3D talking head generation have made significant progress in lip synchronization. However, existing models still struggle to capture the perceptual alignment between varying speech characteristics and corresponding lip movements. In this work, we claim that three criteria -- Temporal Synchronization, Lip Readability, and Expressiveness -- are crucial for achieving perceptually accurate lip movements. Motivated by our hypothesis that a desirable representation space exists to meet these three criteria, we introduce a speech-mesh synchronized representation that captures intricate correspondences between speech signals and 3D face meshes. We found that our learned representation exhibits desirable characteristics, and we plug it into existing models as a perceptual loss to better align lip movements to the given speech. In addition, we utilize this representation as a perceptual metric and introduce two other physically grounded lip synchronization metrics to assess how well the generated 3D talking heads align with these three criteria. Experiments show that training 3D talking head generation models with our perceptual loss significantly improve all three aspects of perceptually accurate lip synchronization. Codes and datasets are available at https://perceptual-3d-talking-head.github.io/.