超對齊：基於超網絡的擴散模型高效測試時對齊技術

摘要

扩散模型虽能实现顶尖性能，却常难以生成符合人类偏好与意图的输出，导致图像美学质量欠佳及语义不一致。现有对齐方法面临艰难权衡：微调方法会因奖励过优化而丧失多样性，测试时缩放方法则带来显著计算开销且易优化不足。为突破这些局限，我们提出HyperAlign——一种训练超网络以实现高效测试时对齐的新框架。该框架不直接修改隐状态，而是动态生成低秩适配权重来调制扩散模型的生成算子，使去噪轨迹能根据输入潜变量、时间步和提示词进行自适应调整，实现奖励条件对齐。我们开发了多种HyperAlign变体，其差异在于超网络的应用频率，以平衡性能与效率。此外，我们采用偏好数据正则化的奖励得分目标来优化超网络，以减少奖励破解现象。在Stable Diffusion和FLUX等扩展生成范式中评估表明，HyperAlign在提升语义一致性与视觉吸引力方面显著优于现有微调及测试时缩放基线方法。

English

Diffusion models achieve state-of-the-art performance but often fail to generate outputs that align with human preferences and intentions, resulting in images with poor aesthetic quality and semantic inconsistencies. Existing alignment methods present a difficult trade-off: fine-tuning approaches suffer from loss of diversity with reward over-optimization, while test-time scaling methods introduce significant computational overhead and tend to under-optimize. To address these limitations, we propose HyperAlign, a novel framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states, HyperAlign dynamically generates low-rank adaptation weights to modulate the diffusion model's generation operators. This allows the denoising trajectory to be adaptively adjusted based on input latents, timesteps and prompts for reward-conditioned alignment. We introduce multiple variants of HyperAlign that differ in how frequently the hypernetwork is applied, balancing between performance and efficiency. Furthermore, we optimize the hypernetwork using a reward score objective regularized with preference data to reduce reward hacking. We evaluate HyperAlign on multiple extended generative paradigms, including Stable Diffusion and FLUX. It significantly outperforms existing fine-tuning and test-time scaling baselines in enhancing semantic consistency and visual appeal.

超對齊：基於超網絡的擴散模型高效測試時對齊技術

HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models

摘要

Support