PonderPounce：一种作为机器人控制回合上下文引擎的预训练多模态大语言模型

摘要

多模态大语言模型（MLLMs）能够整合长时间视觉历史、在部分可观测条件下进行推理，并能从少量示例中推断行为。然而，视觉-语言-动作（VLA）模型通常继承预训练表示，却未将这种上下文能力用作情节记忆。记忆依赖策略通过专门构建的历史机制来弥补这一差距。PonderPounce则复用MLLM原生的因果上下文作为机器人记忆。Ponder作为一个System2 MLLM，在其原生因果上下文中累积情节观测、演示和先前的认知过程，并能生成用于内部使用的子目标文本和演示推理。Pounce作为一个System1 VLA，直接接收当前观测、指令和本体感知；通过Ponder–Pounce接口，它仅异步接收最新的连续认知令牌及其年龄。两者无需专门构建的记忆模块或单独的桥接预训练，即可端到端联合训练。优化后的服务实现认知刷新p50延迟78ms、动作模型调用p50延迟25ms，支持20Hz动作播放。在基于基础规模训练数据的RoboMME上，PonderPounce在相同Pounce架构和接口下，9B模型达到60.83%，0.8B模型达到50.04%，而FrameSamp+Modul为44.51%，当前观测π_{0.5}为17.93%。在9倍数据下，PonderPounce达到75.54%，FrameSamp+Modul为57.88%。在RoboCasa-DC上，同一接口仅依靠动作监督学习即可达到12.5%，而最强已发表的演示条件基线为11.6%；当认知被学习到的空状态替代时，性能下降至8.6%。

English

Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.