IMUSIC：基於IMU的面部表情捕捉

IMUSIC: IMU-based Facial Expression Capture

February 3, 2024

作者: Youjia Wang, Yiwen Wu, Ruiqian Li, Hengan Zhou, Hongyang Lin, Yingwenqi Jiang, Yingsheng Zhu, Guanpeng Long, Jingya Wang, Lan Xu, Jingyi Yu

cs.AI

摘要

在面部動作捕捉和分析方面，主導的解決方案通常基於視覺線索，但這些方法無法保護隱私且容易受遮擋影響。慣性測量單元（IMUs）被視為潛在的解決方案，但主要用於全身動作捕捉。本文提出了IMUSIC來填補這一空白，這是一種使用純IMU信號進行面部表情捕捉的新途徑，與以往的視覺解決方案有顯著差異。我們的IMUSIC設計的關鍵在於三個方面。首先，我們設計微型IMUs以適應面部捕捉，並結合基於解剖學的IMU放置方案。然後，我們提供了一個新穎的IMU-ARKit數據集，為多樣的面部表情和表現提供豐富的配對IMU/視覺信號。這種獨特的多模態為未來方向帶來了巨大潛力，例如基於IMU的面部行為分析。此外，利用IMU-ARKit，我們引入了一種強大的基準方法，可以從純IMU信號準確預測面部混合形狀參數。具體來說，我們為這一新型跟踪任務量身定制了一個Transformer擴散模型，並採用了兩階段訓練策略。IMUSIC框架使我們能夠在視覺方法失敗並同時保護用戶隱私的情況下進行準確的面部捕捉。我們進行了大量關於IMU配置和技術組件的實驗，以驗證我們的IMUSIC方法的有效性。值得注意的是，IMUSIC實現了各種潛在和新穎的應用，例如保護隱私的面部捕捉、對抗遮擋的混合捕捉，或者檢測通常無法通過視覺線索看到的微小面部運動。我們將釋出我們的數據集和實現，以豐富社區中面部捕捉和分析的更多可能性。

English

For facial motion capture and analysis, the dominated solutions are generally based on visual cues, which cannot protect privacy and are vulnerable to occlusions. Inertial measurement units (IMUs) serve as potential rescues yet are mainly adopted for full-body motion capture. In this paper, we propose IMUSIC to fill the gap, a novel path for facial expression capture using purely IMU signals, significantly distant from previous visual solutions.The key design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial capture, companion with an anatomy-driven IMU placement scheme. Then, we contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual signals for diverse facial expressions and performances. Such unique multi-modality brings huge potential for future directions like IMU-based facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong baseline approach to accurately predict facial blendshape parameters from purely IMU signals. Specifically, we tailor a Transformer diffusion model with a two-stage training strategy for this novel tracking task. The IMUSIC framework empowers us to perform accurate facial capture in scenarios where visual methods falter and simultaneously safeguard user privacy. We conduct extensive experiments about both the IMU configuration and technical components to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables various potential and novel applications, i.e., privacy-protecting facial capture, hybrid capture against occlusions, or detecting minute facial movements that are often invisible through visual cues. We will release our dataset and implementations to enrich more possibilities of facial capture and analysis in our community.