SmartMage：面向三維場景理解的動態模態編排

摘要

理解3D場景是具身智能的基礎，這需要在多種模態的異構資訊上進行聯合推理，包括視覺與幾何線索。然而，這些模態的相關性往往會因查詢而異。現有的多模態大語言模型（MLLMs）通常依賴固定的模態組合，忽略了查詢依賴的模態需求。如此僵化的設計可能引入來自無關模態的語義雜訊，同時未能充分利用更具資訊量的模態，導致計算浪費與推理能力被稀釋。為了解決這些挑戰，本文提出SmartMage，一個統一的多模態大語言模型，能夠動態編排異構模態以實現語義感知的3D場景理解。具體而言，SmartMage包含：(1) 一個語義引導的模態自適應路由（SMART）模組，利用語義先驗、文本-模態對齊以及模態品質來選擇與任務相關的模態；以及 (2) 一個模態感知門控專家（MAGE）模組，利用模態先驗引導專家激活，從而在多模態推理中實現自適應的專業化。在實驗上，SmartMage在五個3D場景理解基準上達到了最先進的效能，並在僅RGB的影片理解基準上取得了具競爭力的結果。在我們的診斷基準ScanFacet中，任務被劃分為細粒度的語義類別，使得能夠分析每個語義類型所偏好的模態組合。所觀察到的模態-語義模式進一步提供了SmartMage有效性的證據。專案頁面：https://yuecheong.github.io/SmartMage/。

English

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.