StreamPI：面向视觉-语言-动作模型的流式多模态时间建模

摘要

视觉-语言-动作（VLA）模型已在机器人操作中展现出有效性，然而诸如pi0.5等最先进的模型仍基于单帧范式运行，这限制了其保留过往观测信息和发展精准空间感知的能力。本文提出StreamPI，一种流式多模态时序建模框架，在无需引入任何额外参数的情况下，赋予单帧VLA时序推理能力。其核心设计之一是指令锚定时序建模。该设计将每个（视觉观测，语言指令）对视为一个原子时序单元：单元内的双向注意力实现跨模态融合，而单元间的因果注意力保持自回归流式推理。这确保了语言指令在整个任务执行过程中充当持续的语义锚点。为弥合同步训练与异步真实机器人部署之间的差距，我们引入随机间隔流式训练策略：合适的帧间间隔（例如每3帧）能够实现更快更平滑的动作执行。此外，对间隔进行随机化进一步提升了模型对帧时序扰动的鲁棒性，从而支持实际场景中的异步部署。更进一步，借助大语言模型主干的长度外推能力，StreamPI无缝继承了预训练的单帧权重，并支持灵活的单帧与多帧推理。在涵盖记忆依赖与精准感知场景的真实机器人任务以及仿真基准LIBERO上的实验表明，StreamPI在多样任务上均优于pi0.5。

English

Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.