4D中的人類：使用Transformer 重建和追踪人類

摘要

我們提出了一種重建人類並隨時間追踪他們的方法。在我們方法的核心，我們提出了一個完全「Transformer 化」的人體網絡版本，用於人體網格恢復。這個網絡，HMR 2.0，推進了技術水平，展示了分析過去難以從單張圖像重建的不尋常姿勢的能力。為了分析視頻，我們使用從 HMR 2.0 獲得的 3D 重建作為操作於 3D 空間的追踪系統的輸入。這使我們能夠處理多人情況並通過遮擋事件保持身份。我們的完整方法，4DHumans，在從單眼視頻追踪人們方面實現了最先進的成果。此外，我們展示了 HMR 2.0 在下游動作識別任務上的有效性，相較於以前基於姿勢的動作識別方法，實現了顯著的改進。我們的代碼和模型可在項目網站上找到：https://shubham-goel.github.io/4dhumans/.

English

We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.

4D中的人類：使用Transformer 重建和追踪人類

Humans in 4D: Reconstructing and Tracking Humans with Transformers

摘要

Support