HarmoHOI：調和外觀與三維運動的多視角手物互動合成

摘要

手-物體互動（HOI）合成是動畫製作與具身AI的基石。儘管影片基礎模型具備強大的先驗知識，但由於複雜的手部動作與遮擋，多視角一致的HOI合成仍極具挑戰。我們提出HarmoHOI，這是一個統一的擴散框架，能夠聯合且和諧地生成同步的多視角HOI影片及全域對齊的3D點軌跡。我們的核心洞見在於：穩健的多視角一致性從根本上需要全域對齊的3D幾何與運動。為此，我們提出混合多視角擴散Transformer，共同建模RGB影片與3D點軌跡。透過將點軌跡表示為偽影片，我們將3D幾何訊號與基礎模型的2D潛在空間對齊，從而縮小領域差距並簡化先驗知識的適應。為進一步確保幾何一致性，我們引入全域運動對齊擴散，將粗略的點軌跡優化為公制尺度、全域對齊的3D軌跡。HarmoHOI可在去噪過程中即時共同演化2D外觀與3D運動。為克服多視角HOI數據的稀缺性，我們採用混合數據課程學習策略，成功將通用先驗知識從單視角數據遷移至同步多視角生成。實驗結果顯示，HarmoHOI在視覺品質、運動合理性及多視角幾何一致性方面皆達到最佳表現。專案頁面請見 https://droliven.github.io/HarmoHOI_project。

English

Hand-Object Interaction (HOI) synthesis is a cornerstone for animation production and embodied AI. Despite the strong priors of video foundation models, multi-view consistent HOI synthesis remains challenging due to complex hand motions and occlusions. We present HarmoHOI, a unified diffusion framework that jointly and harmoniously generates synchronized multi-view HOI videos and globally aligned 3D point tracks. Our core insight is that robust multi-view consistency fundamentally requires globally aligned 3D geometry and motion. To this end, we propose a Mixture of Multi-view Diffusion Transformer that co-models RGB videos and 3D point tracks. By representing point tracks as pseudo-videos, we align 3D geometric signals with the 2D latent space of foundation models, thereby minimizing the domain gap and easing adaptation of priors. To further ensure geometry consistency, we introduce Global Motion Aligning Diffusion, which refines coarse point tracks into metric-scale, globally aligned 3D trajectories. HarmoHOI enables on-the-fly co-evolution of 2D appearance and 3D motion during denoising. To overcome the scarcity of multi-view HOI data, we employ a hybrid data curriculum learning strategy that successfully transfers generic priors from single-view data to synchronized multi-view generation. Experimental results show that HarmoHOI achieves state-of-the-art performance in visual quality, motion plausibility, and multi-view geometric consistency. Project page available at https://droliven.github.io/HarmoHOI_project.