从数据不平衡中学习密集手部接触估计

Learning Dense Hand Contact Estimation from Imbalanced Data

May 16, 2025

作者: Daniel Sungho Jung, Kyoung Mu Lee

cs.AI

摘要

手部在人类互动中至关重要，理解手部与世界的接触能促进对其功能的全面认识。近年来，涵盖与物体、另一只手、场景及身体交互的手部互动数据集日益增多。尽管该任务的重要性与日俱增且高质量数据不断积累，如何有效学习密集手部接触估计仍是一个待深入探索的领域。学习密集手部接触估计面临两大挑战：首先，手部接触数据集中存在类别不平衡问题，多数样本未处于接触状态；其次，数据集存在空间分布不均，大部分手部接触集中在指尖，这为推广至手部其他区域的接触带来了困难。为解决这些问题，我们提出了一个从非平衡数据中学习密集手部接触估计（HACO）的框架。针对类别不平衡，我们引入了平衡接触采样法，通过构建并采样多个能公平代表接触与非接触样本多样统计特征的采样组。此外，为应对空间分布不均，我们提出了顶点级类别平衡（VCB）损失函数，该函数通过根据每个顶点在数据集中的接触频率单独重新加权其损失贡献，从而融入了空间变化的接触分布。因此，我们能够有效利用大规模手部接触数据预测密集手部接触估计，而无需担忧类别与空间不平衡问题。相关代码将予以公开。

English

Hands are essential to human interaction, and understanding contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of samples are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact samples. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes will be released.