通過連續的代理到動作學習，在世界空間中實時單眼全身捕捉

Real-time Monocular Full-body Capture in World Space via Sequential Proxy-to-Motion Learning

July 3, 2023

作者: Yuxiang Zhang, Hongwen Zhang, Liangxiao Hu, Hongwei Yi, Shengping Zhang, Yebin Liu

cs.AI

摘要

最近，基於學習的單眼運動捕捉方法已顯示出潛力，通過學習以數據驅動的方式進行回歸。然而，由於數據收集和網絡設計方面的挑戰，現有解決方案仍然難以實現在世界空間中準確的實時全身捕捉。在這項工作中，我們提出了一種順序代理到運動學習方案，以及一個包含2D骨架序列和世界空間中的3D旋轉運動的代理數據集。這樣的代理數據使我們能夠構建一個基於學習的網絡，具有準確的全身監督，同時也減輕了泛化問題。為了更準確和物理合理的預測，我們在我們的網絡中提出了一個考慮接觸的神經運動下降模塊，以便它能夠意識到腳地接觸和與代理觀察的運動不一致。此外，我們在我們的網絡中共享身體-手部上下文信息，以更好地恢復與全身模型相容的手腕姿勢。通過提出的基於學習的解決方案，我們展示了第一個具有世界空間中合理腳地接觸的實時單眼全身捕捉系統。更多視頻結果可在我們的項目頁面找到：https://liuyebin.com/proxycap。

English

Learning-based approaches to monocular motion capture have recently shown promising results by learning to regress in a data-driven manner. However, due to the challenges in data collection and network designs, it remains challenging for existing solutions to achieve real-time full-body capture while being accurate in world space. In this work, we contribute a sequential proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton sequences and 3D rotational motions in world space. Such proxy data enables us to build a learning-based network with accurate full-body supervision while also mitigating the generalization issues. For more accurate and physically plausible predictions, a contact-aware neural motion descent module is proposed in our network so that it can be aware of foot-ground contact and motion misalignment with the proxy observations. Additionally, we share the body-hand context information in our network for more compatible wrist poses recovery with the full-body model. With the proposed learning-based solution, we demonstrate the first real-time monocular full-body capture system with plausible foot-ground contact in world space. More video results can be found at our project page: https://liuyebin.com/proxycap.