通过序列化代理到运动学习实现世界空间中的实时单目全身捕捉

摘要

基于学习的单目运动捕捉方法近期通过数据驱动的回归学习展现出良好效果。然而受限于数据采集与网络设计的挑战，现有方案难以在实现世界坐标系下精准捕捉的同时达到实时全身运动重建。本研究提出了一种序列化代理到动作的学习框架，并构建了包含世界坐标系下二维骨骼序列与三维旋转运动的代理数据集。此类代理数据使我们能够构建具有精确全身监督的学习网络，同时缓解泛化问题。为实现更精准且物理合理的预测，我们在网络中引入了接触感知的神经运动优化模块，使其能够感知足部与地面接触状态以及与代理观测数据的运动偏差。此外，我们通过网络中的身体-手部上下文信息共享，实现了与全身模型更兼容的手腕姿态恢复。凭借所提出的基于学习的解决方案，我们首次实现了具有合理足地接触的世界坐标系实时单目全身运动捕捉系统。更多视频结果请访问项目页面：https://liuyebin.com/proxycap。

English

Learning-based approaches to monocular motion capture have recently shown promising results by learning to regress in a data-driven manner. However, due to the challenges in data collection and network designs, it remains challenging for existing solutions to achieve real-time full-body capture while being accurate in world space. In this work, we contribute a sequential proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton sequences and 3D rotational motions in world space. Such proxy data enables us to build a learning-based network with accurate full-body supervision while also mitigating the generalization issues. For more accurate and physically plausible predictions, a contact-aware neural motion descent module is proposed in our network so that it can be aware of foot-ground contact and motion misalignment with the proxy observations. Additionally, we share the body-hand context information in our network for more compatible wrist poses recovery with the full-body model. With the proposed learning-based solution, we demonstrate the first real-time monocular full-body capture system with plausible foot-ground contact in world space. More video results can be found at our project page: https://liuyebin.com/proxycap.

通过序列化代理到运动学习实现世界空间中的实时单目全身捕捉

Real-time Monocular Full-body Capture in World Space via Sequential Proxy-to-Motion Learning

摘要

Support