AlphaGRPO：通过分解式可验证奖励解锁UMMs中的自我反思多模态生成

AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward

May 12, 2026

作者: Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao

cs.AI

摘要

本文提出AlphaGRPO这一新型框架，将群体相对策略优化（GRPO）应用于自回归扩散统一多模态模型（UMMs），无需冷启动阶段即可增强多模态生成能力。该方法解锁了模型的内在潜力，使其能执行高级推理任务：在推理式文本到图像生成中主动推断用户隐含意图，以及通过自我反思式精炼自主诊断并修正生成结果中的偏差。针对实际多模态生成场景中稳定监督信号缺失的挑战，我们引入了分解可验证奖励（DVReward）。与整体标量奖励不同，DVReward利用大型语言模型（LLM）将复杂用户请求分解为原子化、可验证的语义与质量子问题，再通过通用多模态大模型（MLLM）进行评估，从而提供可靠且可解释的反馈。大量实验表明，AlphaGRPO在GenEval、TIIF-Bench、DPG-Bench及WISE等多模态生成基准上均取得稳健提升，同时在未经编辑任务训练的情况下，于GEdit编辑任务中实现显著性能增长。这些结果验证了本方法的自我反思式强化学习能有效利用模型内在理解能力引导高保真生成。项目主页：https://huangrh99.github.io/AlphaGRPO/

English

In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without an additional cold-start stage. Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation. Project page: https://huangrh99.github.io/AlphaGRPO/