DECO：野外環境中3D人體與場景接觸的密集估計

DECO: Dense Estimation of 3D Human-Scene Contact In The Wild

September 26, 2023

作者: Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, Michael J. Black

cs.AI

摘要

了解人類如何利用身體接觸與世界互動是實現以人為中心的人工智慧的關鍵。推斷3D接觸對於建模逼真且符合物理的人-物互動至關重要，然而現有方法要麼專注於2D，要麼考慮身體關節而非表面，要麼使用粗糙的3D身體區域，或者無法推廣至野外圖像。相反，我們專注於推斷在任意圖像中人體表面與物體之間的密集3D接觸。為了實現這一目標，我們首先收集了一個新的數據集DAMON，其中包含與包含複雜人-物和人-場景接觸的RGB圖像配對的密集頂點級接觸標註。其次，我們訓練了一個新穎的3D接觸檢測器DECO，該檢測器使用基於身體部位和場景上下文的注意力來估計SMPL身體上的頂點級接觸。DECO建立在人類觀察者通過推理有關接觸身體部位、它們與場景物體的接近程度以及周圍場景上下文來識別接觸的洞察力之上。我們對我們的檢測器在DAMON以及RICH和BEHAVE數據集上進行了廣泛評估。我們在所有基準測試中均明顯優於現有的最先進方法。我們還展示了DECO在自然圖像中多樣且具有挑戰性的現實世界人類互動中具有良好的泛化能力。代碼、數據和模型可在https://deco.is.tue.mpg.de獲得。

English

Understanding how humans use physical contact to interact with the world is key to enabling human-centric artificial intelligence. While inferring 3D contact is crucial for modeling realistic and physically-plausible human-object interactions, existing methods either focus on 2D, consider body joints rather than the surface, use coarse 3D body regions, or do not generalize to in-the-wild images. In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images. To achieve this, we first collect DAMON, a new dataset containing dense vertex-level contact annotations paired with RGB images containing complex human-object and human-scene contact. Second, we train DECO, a novel 3D contact detector that uses both body-part-driven and scene-context-driven attention to estimate vertex-level contact on the SMPL body. DECO builds on the insight that human observers recognize contact by reasoning about the contacting body parts, their proximity to scene objects, and the surrounding scene context. We perform extensive evaluations of our detector on DAMON as well as on the RICH and BEHAVE datasets. We significantly outperform existing SOTA methods across all benchmarks. We also show qualitatively that DECO generalizes well to diverse and challenging real-world human interactions in natural images. The code, data, and models are available at https://deco.is.tue.mpg.de.