回读：预训练多模态大语言模型作为文本到图像生成的零样本奖励模型

摘要

本文提出SpectraReward，一种无需训练的奖励函数，可将预训练多模态大语言模型（MLLMs）直接转化为图像生成强化学习的现成奖励模型。不同于要求MLLM评判生成图像或回答分解后的验证问题，SpectraReward通过单次图像条件化的教师强制前向传播，衡量从生成图像中恢复原始提示的准确程度。我们将平均图像条件提示对数似然作为奖励，直接复用MLLM预训练的图像-文本对齐能力，无需偏好标签或奖励模型微调。进一步地，我们引入Self-SpectraReward——针对统一多模态模型的特殊情况，其中策略自身的理解分支作为其生成分支的奖励模型，形成无需外部奖励模型或外部知识的闭环自我改进框架。大量实验验证了SpectraReward的有效性，涵盖两种扩散模型、三种强化学习算法、来自四个MLLM系列（参数规模从4B到235B）的九种奖励MLLM骨干网络，以及五个分布外文本到图像基准测试。结果表明，SpectraReward和Self-SpectraReward均能显著且一致地提升生成性能，并超越此前基于MLLM的奖励训练方法。进一步分析揭示，更大规模的奖励MLLM并非总是更好，而Self-SpectraReward能够匹配甚至超越规模更大的外部奖励模型，这表明奖励-策略对齐是图像生成强化学习的关键因素。项目主页：https://huangrh99.github.io/SpectraReward/

English

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

回读：预训练多模态大语言模型作为文本到图像生成的零样本奖励模型

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

摘要

Support