從不平衡數據中學習密集手部接觸估計

Learning Dense Hand Contact Estimation from Imbalanced Data

May 16, 2025

作者: Daniel Sungho Jung, Kyoung Mu Lee

cs.AI

摘要

手是人類互動的關鍵，理解手與世界之間的接觸有助於全面認識其功能。近年來，涵蓋與物體、另一隻手、場景及身體互動的手部交互數據集日益增多。儘管這一任務的重要性與日俱增，且高質量數據不斷積累，如何有效學習密集手部接觸估計仍是一個尚未充分探索的領域。學習密集手部接觸估計面臨兩大挑戰：首先，手部接觸數據集中存在類別不平衡問題，大多數樣本並未發生接觸；其次，這些數據集還存在空間分佈不均的問題，大部分手部接觸集中於指尖，這導致對其他手部區域接觸的泛化能力面臨挑戰。為解決這些問題，我們提出了一個從不平衡數據中學習密集手部接觸估計（HACO）的框架。針對類別不平衡問題，我們引入了平衡接觸採樣方法，該方法構建並從多個採樣組中抽取樣本，這些採樣組公平地代表了接觸與非接觸樣本的多樣化統計特性。此外，為應對空間不平衡問題，我們提出了頂點級別類別平衡（VCB）損失函數，它通過根據每個頂點在數據集中的接觸頻率單獨重新加權其損失貢獻，從而納入了空間變化的接觸分佈。因此，我們能夠在不受到類別和空間不平衡問題困擾的情況下，利用大規模手部接觸數據有效預測密集手部接觸估計。相關代碼將被公開。

English

Hands are essential to human interaction, and understanding contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of samples are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact samples. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes will be released.