AffordBot：基于多模态大语言模型的3D细粒度具身推理系统

AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models

November 13, 2025

作者: Xinyi Wang, Xun Yang, Yanlong Xu, Yuchen Wu, Zhen Li, Na Zhao

cs.AI

摘要

在物理环境中实现有效的人机协作，不仅需要理解动作对象，还需明确可操作元素的空间位置及交互方式。现有方法多停留在物体层面，或割裂地处理细粒度可供性推理，缺乏连贯的指令驱动式 grounding 与推理机制。本研究提出"细粒度三维具身推理"新任务，要求智能体根据任务指令，为三维场景中每个被引用的可供性元素预测包含空间位置、运动类型与运动轴的结构化三元组。为解决该任务，我们提出 AffordBot 创新框架，将多模态大语言模型（MLLMs）与定制化的思维链（CoT）推理范式相结合。为弥合三维输入与二维兼容 MLLMs 之间的鸿沟，我们渲染场景环视图像并将三维候选元素投影至这些视图，形成与场景几何对齐的丰富视觉表征。我们的 CoT 流程始于主动感知阶段，引导 MLLM 根据指令选择最具信息量的视角，继而通过逐步推理定位可供性元素并推断合理的交互运动。在 SceneFun3D 数据集上的评估表明，AffordBot 仅凭三维点云输入和 MLLMs 就实现了最先进的性能，展现出强大的泛化能力与物理接地推理能力。

English

Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate at the object level or disjointedly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce a new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in a 3D scene, a structured triplet comprising its spatial location, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMs) with a tailored chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 2D-compatible MLLMs, we render surround-view images of the scene and project 3D element candidates into these views, forming a rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves state-of-the-art performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs.