RewardSDS：通過獎勵加權採樣實現分數蒸餾對齊

RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling

March 12, 2025

作者: Itay Chachy, Guy Yariv, Sagie Benaim

cs.AI

摘要

分數蒸餾採樣（Score Distillation Sampling, SDS）已成為一種有效技術，用於利用二維擴散先驗來完成諸如文本到三維生成等任務。儘管功能強大，SDS在實現與用戶意圖的精細對齊方面仍存在困難。為克服這一挑戰，我們提出了RewardSDS，這是一種新穎的方法，它根據獎勵模型的對齊分數對噪聲樣本進行加權，從而產生加權的SDS損失。這種損失優先考慮那些能產生對齊且高獎勵輸出的噪聲樣本的梯度。我們的方法具有廣泛的適用性，並能擴展基於SDS的方法。特別是，我們通過引入RewardVSD展示了其在變分分數蒸餾（Variational Score Distillation, VSD）中的應用。我們在文本到圖像、二維編輯以及文本到三維生成任務上評估了RewardSDS和RewardVSD，結果顯示在衡量生成質量及與期望獎勵模型對齊的多樣化指標上，相較於SDS和VSD均有顯著提升，實現了最先進的性能。項目頁面可訪問：https://itaychachy.github.io/reward-sds/。

English

Score Distillation Sampling (SDS) has emerged as an effective technique for leveraging 2D diffusion priors for tasks such as text-to-3D generation. While powerful, SDS struggles with achieving fine-grained alignment to user intent. To overcome this, we introduce RewardSDS, a novel approach that weights noise samples based on alignment scores from a reward model, producing a weighted SDS loss. This loss prioritizes gradients from noise samples that yield aligned high-reward output. Our approach is broadly applicable and can extend SDS-based methods. In particular, we demonstrate its applicability to Variational Score Distillation (VSD) by introducing RewardVSD. We evaluate RewardSDS and RewardVSD on text-to-image, 2D editing, and text-to-3D generation tasks, showing significant improvements over SDS and VSD on a diverse set of metrics measuring generation quality and alignment to desired reward models, enabling state-of-the-art performance. Project page is available at https://itaychachy. github.io/reward-sds/.