超对齐：扩散模型高效测试时对齐的超网络方法

摘要

扩散模型虽能实现最先进的性能，却常难以生成符合人类偏好与意图的输出，导致图像存在审美质量不佳和语义不一致的问题。现有对齐方法面临艰难权衡：微调方法因奖励过优化而丧失多样性，测试时缩放方法则带来显著计算开销且容易优化不足。为突破这些局限，我们提出HyperAlign框架，通过训练超网络实现高效且有效的测试时对齐。该框架不直接修改隐状态，而是动态生成低秩适配权重来调制扩散模型的生成算子，使去噪轨迹能根据输入隐变量、时间步和提示词进行自适应调整，实现奖励条件对齐。我们开发了多种HyperAlign变体，其差异在于超网络的应用频率，以平衡性能与效率。此外，我们采用偏好数据正则化的奖励分数目标来优化超网络，以减少奖励破解现象。在Stable Diffusion和FLUX等扩展生成范式上的实验表明，HyperAlign在提升语义一致性与视觉吸引力方面显著优于现有微调及测试时缩放基线方法。

English

Diffusion models achieve state-of-the-art performance but often fail to generate outputs that align with human preferences and intentions, resulting in images with poor aesthetic quality and semantic inconsistencies. Existing alignment methods present a difficult trade-off: fine-tuning approaches suffer from loss of diversity with reward over-optimization, while test-time scaling methods introduce significant computational overhead and tend to under-optimize. To address these limitations, we propose HyperAlign, a novel framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states, HyperAlign dynamically generates low-rank adaptation weights to modulate the diffusion model's generation operators. This allows the denoising trajectory to be adaptively adjusted based on input latents, timesteps and prompts for reward-conditioned alignment. We introduce multiple variants of HyperAlign that differ in how frequently the hypernetwork is applied, balancing between performance and efficiency. Furthermore, we optimize the hypernetwork using a reward score objective regularized with preference data to reduce reward hacking. We evaluate HyperAlign on multiple extended generative paradigms, including Stable Diffusion and FLUX. It significantly outperforms existing fine-tuning and test-time scaling baselines in enhancing semantic consistency and visual appeal.

超对齐：扩散模型高效测试时对齐的超网络方法

HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models

摘要

Support