Stream3D-VLM: 漸次的幾何事前知識を用いたオンライン3D空間理解

要旨

尽管3D场景理解取得了进展，现有的3D大多模态模型仍运行在离线设置中，需要完整的场景观测或预定义的视频片段。本文提出了一种在线3D视觉语言模型，能够从流式视频中实现实时空间理解。我们的方法采用基于LLM下一个词元预测目标的自回归流控制建模，以学习何时响应，并运用轻量级的视觉-空间特征融合（VSFI）模块，将时间对齐的几何先验逐步注入视觉流中。为缓解长上下文解码开销，我们提出了即插即用的几何自适应体素压缩（GAVC）模块，用于高效的视觉词元压缩。针对流式3D语言数据的稀缺问题，我们进一步开发了一套可扩展的数据生成流程，构建了超过100万个在线时空3D问答对，并建立了包含29项任务的综合基准。大量实验表明，我们的方法在在线和离线的3D空间理解、推理及定位任务上显著优于专有模型和开源模型。项目页面访问地址为 https://stream3d-vlm.github.io/ 。

English

Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/