回讀：預訓練多模態大語言模型是文本到圖像生成的零樣本獎勵模型

摘要

本文提出 SpectraReward，一种无需训练的奖励函数，可将预训练多模态大语言模型（MLLM）直接转化为图像生成强化学习的现成奖励模型。不同于要求MLLM直接评判生成图像或回答分解后的验证问题，SpectraReward通过一次图像条件下的教师强制前向传播，衡量原始提示词能从生成图像中恢复的程度。我们采用平均图像条件提示词对数似然作为奖励，直接复用MLLM预训练的图像-文本对齐能力，无需偏好标签或奖励模型微调。我们进一步引入 Self-SpectraReward——统一多模态模型的特殊情形，其中策略自身理解分支充当其生成分支的奖励模型，构成无需外部奖励模型或外部知识的闭环自我改进框架。大量实验验证了SpectraReward在广泛的图像生成强化学习研究中的有效性，涵盖两种扩散模型、三种强化学习算法、来自四个MLLM系列（参数规模从4B到235B）的九个奖励MLLM骨干网络，以及五个分布外文本到图像基准。结果表明，SpectraReward与Self-SpectraReward均能显著且稳定地提升生成性能，并超越先前基于MLLM的奖励训练方法。进一步分析发现，更大的奖励MLLM并非总是更好，而Self-SpectraReward能够匹配甚至超越更大的外部奖励模型，表明奖励-策略对齐是图像生成强化学习的关键因素。项目主页：https://huangrh99.github.io/SpectraReward/

English

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

回讀：預訓練多模態大語言模型是文本到圖像生成的零樣本獎勵模型

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

摘要

Support