用於真實世界動作語言模型的即插即用2D動作介面

摘要

動作語言模型（MoLMs）通常透過將3D動作進行token化，並使用語言模型處理所得的token來理解人類動作。然而，從單目影片中獲取精確的3D動作具有挑戰性，限制了其在現實世界中的適用性。為了解決此問題，我們引入了一個即插即用的2D動作介面，使預訓練於3D的MoLMs能夠接受2D動作輸入，而無需修改或微調原始模型。在公開資料集上的實驗表明，我們的方法在多個MoLMs上實現了與3D動作輸入相當的性能，並且優於在2D動作上從頭訓練的MoLMs。我們進一步構建了一個單目真實世界影片動作評估資料集，並引入了一個真實影片適配器，證明了在所評估的單目姿態估計設定下，2D動作相較於3D動作的實用性。這些結果表明，2D動作為在現實世界的動作理解場景中部署MoLMs提供了一個實用的介面。程式碼可在 https://github.com/irajisamurai/2D-Motion-Interface 取得。

English

Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.