PonderPounce：预训练多模态大語言模型作為機器人控制的回合上下文引擎

摘要

多模態大型語言模型（MLLMs）能夠整合長時間的視覺歷史、在部分可觀測性下進行推理，並從少數範例中推斷行為。然而，視覺-語言-動作（VLA）模型通常繼承預訓練表徵，卻未將此上下文能力用作情節記憶。依賴記憶的策略透過專門設計的歷史機制來彌補此差距。PonderPounce 則直接重用 MLLM 原生的因果上下文作為機器人記憶。Ponder 是一種 System2 MLLM，在其原生因果上下文中累積情節觀察、示範與先前的認知，並可生成子目標文字與示範推理供內部使用。Pounce 是一種 System1 VLA，直接接收當前觀察、指令與本體感覺；透過 Ponder–Pounce 介面，它僅非同步接收最新的連續認知token及其年齡。兩者以端到端方式聯合訓練，無需專門設計的記憶模組或單獨的橋接預訓練。優化後的服務達到 78ms 的 p50 認知更新延遲與 25ms 的動作模型調用延遲，支援 20Hz 的動作播放。在採用基礎規模訓練資料的 RoboMME 上，PonderPounce 在相同的 Pounce 架構與介面下達到 60.83%（9B）與 50.04%（0.8B），而 FrameSamp+Modul 為 44.51%，僅觀察當前畫面的 π₀.₅ 為 17.93%。在 9 倍資料下，PonderPounce 達到 75.54%，FrameSamp+Modul 為 57.88%。在 RoboCasa-DC 上，相同介面僅從動作監督學習，達到 12.5%，而最強的已發布示範條件基線為 11.6%；當認知被學習到的空狀態取代時，降至 8.6%。

English

Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.