StreamPI：面向視覺-語言-動作模型的流式多模態時序建模

摘要

視覺-語言-動作（VLA）模型已展現出在機器人操作中的有效性，然而諸如 pi0.5 等最先進模型仍以單幀範式運作，限制了其保留過往觀測與發展精確空間感知的能力。在本文中，我們提出 StreamPI，一種串流多模態時序建模框架，在不引入任何額外參數的情況下，賦予單幀 VLA 時序推理能力。核心設計之一是指令錨定的時序建模。該方法將每個（視覺觀測，語言指令）配對視為一個原子時序單元：配對內的雙向注意力實現跨模態融合，而配對間的因果注意力則保持自回歸串流推論。這確保了語言指令在整個任務執行過程中作為持續的語義錨點。為縮小同步訓練與非同步真實機器人部署之間的差距，我們引入隨機間隔串流訓練策略：適當的幀間間隔（例如每 3 幀）能實現更快、更平滑的動作執行。此外，隨機化間隔進一步增強了對幀時序擾動的穩健性，支援實際應用中的非同步部署。更進一步，透過利用 LLM 骨幹的長度外推能力，StreamPI 無縫繼承預訓練的單幀權重，並支援靈活的單幀與多幀推論。涵蓋記憶依賴與精確感知場景的真實機器人任務實驗，以及模擬基準 LIBERO 上的實驗，均表明 StreamPI 在多樣化任務上優於 pi0.5。

English

Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.