SmartMage：面向3D场景理解的动态模态编排

摘要

理解3D场景是具身智能的基础，需要对来自多种模态的异构信息进行联合推理，包括视觉和几何线索。然而，这些模态的相关性往往随查询而变化。现有的多模态大语言模型（MLLMs）通常依赖固定的模态组合，忽视了查询相关的模态需求。这种刚性设计可能引入来自无关模态的语义噪声，同时未能充分利用信息量更大的模态，导致计算浪费和推理效果削弱。为解决这些挑战，本文提出SmartMage——一个统一的多模态大语言模型，能够动态编排异构模态，实现语义感知的3D场景理解。具体而言，SmartMage包含：(1) 语义引导的模态自适应路由（SMART）模块，利用语义先验、文本-模态对齐和模态质量来选择任务相关模态；(2) 模态感知门控专家（MAGE）模块，利用模态先验引导专家激活，促进多模态推理中的自适应专门化。实验结果表明，SmartMage在五个3D场景理解基准上取得了最先进的性能，并在仅RGB视频理解基准上获得了具有竞争力的结果。在我们的诊断基准ScanFacet中，任务被划分为细粒度的语义类别，从而能够分析每种语义类型偏好的模态组合。观察到的模态-语义模式进一步证明了SmartMage的有效性。项目页面：https://yuecheong.github.io/SmartMage/。

English

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.