Stream3D-VLM：基於增量幾何先驗的線上3D空間理解

摘要

儘管3D場景理解技術有所進展，現有的3D大型多模態模型仍僅適用於離線設定，需完整的場景觀測或預先定義的影片片段。本文提出一種線上3D視覺-語言模型，能從串流影片中實現即時空間理解。我們的方法採用基於大型語言模型下一個詞元預測目標的自迴歸串流控制建模，以學習何時應回應，並運用輕量級的視覺-空間特徵整合模組，逐步將時間對齊的幾何先驗注入視覺流中。為減輕長上下文解碼的計算負擔，我們提出即插即用的幾何自適應體素壓縮模組，以實現高效的視覺詞元壓縮。針對串流3D語言資料匱乏的問題，我們進一步開發可擴展的資料生成流程，整理出超過100萬組線上空時3D問答對，並建立涵蓋29項任務的全面基準測試。大量實驗證明，我們的方法在線上和離線的3D空間理解、推理及定位任務中，均顯著優於專有模型與開源模型。專案頁面位於 https://stream3d-vlm.github.io/

English

Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/