AlphaGRPO：透過分解可驗證獎勵解鎖UMMs中的自反多模態生成

AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward

May 12, 2026

作者: Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao

cs.AI

摘要

本文提出AlphaGRPO，一个创新框架，將群體相對策略最佳化（GRPO）應用於AR-Diffusion統一多模態模型（UMMs），在不需額外冷啟動階段的情況下增強多模態生成能力。我們的方法釋放模型內在潛能，使其能執行進階推理任務：推理式文字到圖像生成——模型主動推斷使用者隱含意圖；以及自我反思式精煉——模型自主診斷並修正生成輸出中的偏差。為解決在真實多模態生成場景中提供穩定監督的挑戰，我們引入分解式可驗證獎勵（DVReward）。不同於整體標量獎勵，DVReward利用大型語言模型（LLM）將複雜使用者請求分解為原子式、可驗證的語義與品質問題，再由通用多模態大型語言模型（MLLM）進行評估，提供可靠且可解釋的回饋。大量實驗顯示，AlphaGRPO在多模態生成基準（包括GenEval、TIIF-Bench、DPG-Bench及WISE）上帶來穩健提升，且在未經編輯任務訓練的情況下，於GEdit編輯任務中亦取得顯著進展。這些結果驗證了我們的自我反思式強化方法能有效利用既有理解能力，引導高保真生成。專案頁面：https://huangrh99.github.io/AlphaGRPO/

English

In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without an additional cold-start stage. Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation. Project page: https://huangrh99.github.io/AlphaGRPO/