将交织多模态推理桥接为统一决策过程

摘要

统一多模态模型（UMMs）已展现出令人瞩目的图文交错推理能力，然而通过强化学习（RL）有效优化此类多轮生成仍是一个开放挑战。现有方法仅将RL应用于文本步骤，而将图像生成交由监督代理处理，导致策略梯度无法在跨异质模态的完整交错轨迹中传播，这使得RL在UMMs中的潜力远未得到充分挖掘。本文提出BRAID（将交错多模态推理桥接为统一决策过程）——一个简洁框架，它将多轮图文推理建模为统一马尔可夫决策过程（MDP），从而通过单一、原则性的RL目标联合优化文本与视觉生成。BRAID计算共享的轨迹级优势函数，并将其一致地传播至文本令牌和图像去噪路径，每条路径通过其模态原生策略梯度机制进行优化。为应对长期信用分配问题，BRAID进一步采用视觉语言模型（VLM）裁判，对每个中间图像依据其推理效用进行评分，从而在关键视觉分支处提供密集的轮次级反馈以强化学习效果。在空间推理与视觉感知基准上的实验表明，BRAID持续优于多种基线方法，证实了融合视觉思维引导的统一MDP公式对于实现高效多模态推理至关重要。

English

Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge. Existing approaches apply RL exclusively to text steps, relegating image generation to supervised surrogates, preventing policy gradients from propagating through the full interleaved trajectory across heterogeneous modalities. This leaves the potential of RL for UMMs largely untapped. In the paper, we introduce BRAID (Bridging inteRleAved multI-modal reasoning as a unified Decision process), a simple framework that casts multi-turn text-image-text reasoning as a unified Markov decision process (MDP), enabling joint optimization of textual and visual generation via a single, principled RL objective. BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths, each optimized through its modality-native policy gradient mechanism. To further address long-horizon credit assignment, BRAID employs a vision-language model (VLM) judge that scores each intermediate image on its reasoning utility, supplying dense turn-level feedback to sharpen learning at critical visual branches. Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines, confirming that a unified MDP formulation with vision-thinking guidance is essential for effective multi-modal reasoning.