双流扩散模型赋能世界模型增强的视觉-语言-动作系统

Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model

October 31, 2025

作者: John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin

cs.AI

摘要

近期，通过引入世界模型增强视觉-语言-动作模型（VLA）在机器人策略学习方面展现出潜力。然而，由于状态观测与动作序列两种模态间的固有差异，联合预测下一状态观测和动作序列仍具挑战性。为此，我们提出双流扩散框架（DUST），这一世界模型增强型VLA框架通过处理模态冲突，有效提升了模型在多样化任务中的性能。具体而言，我们设计了一种多模态扩散Transformer架构，在保持独立模态流的同时实现跨模态知识共享。此外，我们引入了针对各模态的独立噪声扰动机制和解耦流匹配损失函数。该设计使模型能够以双向方式学习联合分布，同时避免了对统一潜在空间的需求。基于训练阶段的模态解耦，我们还提出了支持测试时缩放的交联合采样方法，使动作与视觉令牌能够以不同速率异步演化。在RoboCasa和GR-1等模拟基准测试中，DUST相较基线方法最高提升6%的性能，而测试时缩放策略额外带来2-5%的增益。在基于Franka Research 3的真实任务中，DUST将成功率提高13%，证实了其超越仿真环境的有效性。此外，在BridgeV2无动作视频数据集上的预训练为RoboCasa任务带来显著迁移增益，凸显了DUST在大规模VLA预训练方面的潜力。

English

Recently, augmenting Vision-Language-Action models (VLAs) with world modeling has shown promise in improving robotic policy learning. However, it remains challenging to jointly predict next-state observations and action sequences because of the inherent difference between the two modalities. To address this, we propose DUal-STream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict and enhances the performance of VLAs across diverse tasks. Specifically, we propose a multimodal diffusion transformer architecture that explicitly maintains separate modality streams while still enabling cross-modal knowledge sharing. In addition, we introduce independent noise perturbations for each modality and a decoupled flow-matching loss. This design enables the model to learn the joint distribution in a bidirectional manner while avoiding the need for a unified latent space. Based on the decoupling of modalities during training, we also introduce a joint sampling method that supports test-time scaling, where action and vision tokens evolve asynchronously at different rates. Through experiments on simulated benchmarks such as RoboCasa and GR-1, DUST achieves up to 6% gains over baseline methods, while our test-time scaling approach provides an additional 2-5% boost. On real-world tasks with the Franka Research 3, DUST improves success rates by 13%, confirming its effectiveness beyond simulation. Furthermore, pre-training on action-free videos from BridgeV2 yields significant transfer gains on RoboCasa, underscoring DUST's potential for large-scale VLA pretraining.