噪音校準：使用預先訓練的視頻擴散模型進行即插即用的保留內容的視頻增強

Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models

July 14, 2024

作者: Qinyu Yang, Haoxin Chen, Yong Zhang, Menghan Xia, Xiaodong Cun, Zhixun Su, Ying Shan

cs.AI

摘要

為了提升合成影片的品質，目前主要的方法之一是重新訓練專家擴散模型，然後實施一個噪聲去噪過程進行精煉。儘管訓練成本高昂，但在原始影片與增強影片之間保持內容一致性仍然是一個重大挑戰。為了應對這一挑戰，我們提出了一種新穎的公式，同時考慮視覺品質和內容一致性。通過一個提出的損失函數來確保內容的一致性，該函數保持輸入的結構，同時通過利用預訓練擴散模型的去噪過程來提高視覺品質。為了解決所制定的優化問題，我們開發了一種即插即用的噪聲優化策略，稱為噪聲校準。通過通過幾次迭代來精煉初始隨機噪聲，可以在很大程度上保留原始影片的內容，並且增強效果表現出顯著的改善。大量實驗證明了所提方法的有效性。

English

In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation that considers both visual quality and consistency of content. Consistency of content is ensured by a proposed loss function that maintains the structure of the input, while visual quality is improved by utilizing the denoising process of pretrained diffusion models. To address the formulated optimization problem, we have developed a plug-and-play noise optimization strategy, referred to as Noise Calibration. By refining the initial random noise through a few iterations, the content of original video can be largely preserved, and the enhancement effect demonstrates a notable improvement. Extensive experiments have demonstrated the effectiveness of the proposed method.