DECO：野外环境中3D人体与场景接触的密集估计

DECO: Dense Estimation of 3D Human-Scene Contact In The Wild

September 26, 2023

作者: Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, Michael J. Black

cs.AI

摘要

理解人类如何利用身体接触与世界互动对于实现以人为中心的人工智能至关重要。推断3D接触对于建模逼真且符合物理规律的人-物互动至关重要，然而现有方法要么专注于2D，要么考虑身体关节而非表面，要么使用粗糙的3D身体区域，要么无法推广到野外图像。相反，我们专注于推断在任意图像中人体表面与物体之间的密集3D接触。为实现这一目标，我们首先收集了DAMON，一个新的数据集，其中包含与包含复杂人-物和人-场景接触的RGB图像配对的密集顶点级接触注释。其次，我们训练了DECO，一种新颖的3D接触检测器，它利用基于身体部位驱动和场景上下文驱动的注意力来估计SMPL身体上的顶点级接触。DECO建立在这样一个洞察力之上，即人类观察者通过推理接触的身体部位、它们与场景物体的接近程度以及周围场景上下文来识别接触。我们在DAMON以及RICH和BEHAVE数据集上对我们的检测器进行了广泛评估。我们在所有基准测试中明显优于现有的SOTA方法。我们还定性展示DECO在自然图像中广泛且具有挑战性的真实世界人类互动中具有良好的泛化能力。代码、数据和模型可在https://deco.is.tue.mpg.de获取。

English

Understanding how humans use physical contact to interact with the world is key to enabling human-centric artificial intelligence. While inferring 3D contact is crucial for modeling realistic and physically-plausible human-object interactions, existing methods either focus on 2D, consider body joints rather than the surface, use coarse 3D body regions, or do not generalize to in-the-wild images. In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images. To achieve this, we first collect DAMON, a new dataset containing dense vertex-level contact annotations paired with RGB images containing complex human-object and human-scene contact. Second, we train DECO, a novel 3D contact detector that uses both body-part-driven and scene-context-driven attention to estimate vertex-level contact on the SMPL body. DECO builds on the insight that human observers recognize contact by reasoning about the contacting body parts, their proximity to scene objects, and the surrounding scene context. We perform extensive evaluations of our detector on DAMON as well as on the RICH and BEHAVE datasets. We significantly outperform existing SOTA methods across all benchmarks. We also show qualitatively that DECO generalizes well to diverse and challenging real-world human interactions in natural images. The code, data, and models are available at https://deco.is.tue.mpg.de.