打破故障级联：面向医学多模态推理的步骤感知强化学习

摘要

近期多模态大语言模型在临床图像推理领域展现出巨大潜力，但现有后训练流程仍主要采用以结果为中心的模式，仅依赖最终答案正确性或序列级偏好进行优化。这种方案面临稀疏信用分配问题，难以优化对临床应用至关重要的推理过程。我们的分析表明，由早期推理故障引发的级联错误是导致医学视觉问答（VQA）基准测试中预测错误的主要原因。基于此，我们提出医学推理感知策略优化（MRPO），这是一种融合逐步过程奖励的强化学习算法。当最终答案错误时，MRPO会对早期无效推理步骤中的标记施加指数级增大的惩罚，从而打破故障级联而不影响正确路径。在三个多模态大语言模型骨干网络上，MRPO均显著优于标准GRPO及近期强化学习基线，其中Qwen3-VL-8B-Instruct模型甚至超越规模更大的医学多模态大语言模型如HuatuoGPT-Vision-34B（领先2.79个百分点）。此外，MRPO将早期推理故障率从64.0%降至13.0%，表明针对级联故障的定向缓解不仅能提升推理质量，还能提高最终答案准确性。我们的代码已开源至 https://github.com/dmis-lab/MRPO

English

Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO