用于现实世界运动语言模型的即插即用2D运动接口

摘要

运动语言模型（MoLMs）通常通过将三维运动词元化，并利用语言模型处理生成的词元来理解人体运动。然而，从单目视频中获取准确的三维运动具有挑战性，限制了其在实际场景中的应用。为解决这一问题，我们引入了一种即插即用的二维运动接口，使得基于三维预训练的MoLMs能够接受二维运动输入，而无需修改或微调原始模型。在公开数据集上的实验表明，我们的方法在多种MoLMs上取得了与三维运动输入相当的性能，并且优于在二维运动上从头训练的MoLMs。我们进一步构建了一个单目真实世界视频运动评估数据集，并引入了一个真实视频适配器，证明了在所评估的单目姿态估计设置下，二维运动相较于三维运动更具实用性。这些结果表明，二维运动为MoLMs在真实运动理解场景中的部署提供了一种实用接口。代码已开源：https://github.com/irajisamurai/2D-Motion-Interface。

English

Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.