打破失敗級聯：步驟感知強化學習於醫學多模態推理

摘要

近期多模态大型語言模型在臨床影像推理方面展現出巨大潛力，但現有的後訓練流程仍主要以結果為中心，依賴於最終答案的正確性或序列層級偏好。這種做法存在稀疏信用分配問題，導致難以優化對臨床應用至關重要的推理過程。我們的分析顯示，來自早期推理失敗的級聯錯誤是醫學視覺問答（VQA）基準中預測錯誤的主要原因。基於此，我們提出醫學推理感知策略優化（MRPO），這是一種整合逐步過程獎勵的強化學習演算法。當最終答案錯誤時，MRPO會對早期無效推理步驟中的詞元施加指數級增大的懲罰，從而在不影響成功路徑的情況下打破失敗級聯。在三種多模態LLM骨幹架構上，MRPO持續優於標準GRPO及近期基線強化學習方法；在Qwen3-VL-8B-Instruct上，甚至超越規模顯著更大的醫學多模態語言模型（如HuatuoGPT-Vision-34B）2.79個百分點。此外，MRPO將早期推理失敗率從64.0%降至13.0%，顯示針對性緩解級聯失敗既能提升推理品質，也能提高最終答案準確率。我們的程式碼開源於 https://github.com/dmis-lab/MRPO。

English

Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO