4D中的人类：使用Transformer重建和跟踪人类

摘要

我们提出了一种重建人类并随时间跟踪他们的方法。在我们的方法的核心，我们提出了一个完全“Transformer化”的人体网格恢复网络版本。这个网络，HMR 2.0，推动了技术发展，并展示了分析过去难以从单个图像重建的不寻常姿势的能力。为了分析视频，我们使用来自HMR 2.0的3D重建作为3D操作的跟踪系统的输入。这使我们能够处理多人并通过遮挡事件保持身份。我们的完整方法，4DHumans，在从单目视频跟踪人员方面取得了最先进的结果。此外，我们展示了HMR 2.0在动作识别下游任务上的有效性，相比之前基于姿势的动作识别方法，取得了显著的改进。我们的代码和模型可在项目网站上找到：https://shubham-goel.github.io/4dhumans/.

English

We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.

4D中的人类：使用Transformer重建和跟踪人类

Humans in 4D: Reconstructing and Tracking Humans with Transformers

摘要

Support