ArtHOI：利用基础模型实现手部-物体交互的单目4D重建驯化研究

ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions

March 26, 2026

作者: Zikai Wang, Zhilu Zhang, Yiqing Wang, Hui Li, Wangmeng Zuo

cs.AI

摘要

现有的人-物交互方法大多局限于刚性物体，而针对铰接物体的四维重建方法通常需要预先扫描物体甚至使用多视角视频。从单目RGB视频重建铰接物体与人交互的四维动态场景，仍是一个尚未解决但极具意义的挑战。值得庆幸的是，基础模型的近期进展为解决这一高度不适定问题提供了新机遇。为此，我们提出ArtHOI——一个基于优化的框架，能够整合并精炼来自多个基础模型的先验知识。我们的核心贡献在于开发了一套新颖方法体系，专门用于解决这些先验知识固有的不精确性和物理失真问题。具体而言，我们提出了自适应采样优化方法，通过优化物体的度量尺度和位姿，实现其归一化网格在世界空间中的准确定位。此外，我们提出基于多模态大语言模型引导的 hand-object 对齐方法，利用接触关系推理信息作为手-物网格组合优化的约束条件。为进行全面评估，我们还构建了两个新数据集ArtHOI-RGBD和ArtHOI-Wild。大量实验验证了我们的ArtHOI在不同物体和交互场景下的鲁棒性与有效性。项目地址：https://arthoi-reconstruction.github.io。

English

Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.