人机交互中的具身指代表达理解

摘要

随着机器人进入人类工作空间，亟需使其能够理解具身化的人类指令，从而实现直观流畅的人机交互。然而，由于缺乏能够捕捉多样化人机交互场景中自然具身互动的大规模数据集，精确理解仍面临挑战。现有数据集普遍存在视角偏差、单视角采集、非语言手势覆盖不足以及主要聚焦室内环境等问题。为解决这些局限性，我们提出Refer360数据集——一个在室内外多场景下通过多视角采集的大规模具身化语言与非语言交互数据集。此外，我们设计了多模态引导残差模块MuRes，该模块通过构建信息瓶颈提取显著的模态特异性信号，并将其强化注入预训练表征，从而为下游任务构建互补特征。我们在包括Refer360在内的四个人机交互数据集上进行了广泛实验，结果表明当前多模态模型难以全面捕捉具身互动，但通过MuRes增强后能持续提升性能。这些发现确立了Refer360作为重要基准数据集的价值，同时展现出引导残差学习在提升人类环境中机器人的具身指代表达理解能力方面的潜力。

English

As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.

人机交互中的具身指代表达理解

Embodied Referring Expression Comprehension in Human-Robot Interaction

摘要

Support