噪声校准：使用预训练视频扩散模型进行即插即用的保留内容视频增强

Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models

July 14, 2024

作者: Qinyu Yang, Haoxin Chen, Yong Zhang, Menghan Xia, Xiaodong Cun, Zhixun Su, Ying Shan

cs.AI

摘要

为了提高合成视频的质量，目前一种主要的方法涉及重新训练专家扩散模型，然后实施噪声去噪过程进行细化。尽管训练成本高昂，但在原始视频和增强视频之间保持内容一致性仍然是一个重大挑战。为了解决这一挑战，我们提出了一种考虑视觉质量和内容一致性的新颖公式。内容一致性通过一个保持输入结构的损失函数来确保，而视觉质量则通过利用预训练扩散模型的去噪过程来改善。为了解决所制定的优化问题，我们开发了一种即插即用的噪声优化策略，称为噪声校准。通过通过几次迭代对初始随机噪声进行细化，可以在很大程度上保留原始视频的内容，并且增强效果表现出显著改善。大量实验证明了所提方法的有效性。

English

In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation that considers both visual quality and consistency of content. Consistency of content is ensured by a proposed loss function that maintains the structure of the input, while visual quality is improved by utilizing the denoising process of pretrained diffusion models. To address the formulated optimization problem, we have developed a plug-and-play noise optimization strategy, referred to as Noise Calibration. By refining the initial random noise through a few iterations, the content of original video can be largely preserved, and the enhancement effect demonstrates a notable improvement. Extensive experiments have demonstrated the effectiveness of the proposed method.