具有隐藏对齐的视频到音频生成

Video-to-Audio Generation with Hidden Alignment

July 10, 2024

作者: Manjie Xu, Chenxing Li, Yong Ren, Rilin Chen, Yu Gu, Wei Liang, Dong Yu

cs.AI

摘要

随着文本到视频生成领域的显著突破，根据视频输入生成在语义和时间上对齐的音频内容已成为研究人员的关注焦点。在这项工作中，我们旨在深入探讨视频到音频生成范式，重点关注三个关键方面：视觉编码器、辅助嵌入和数据增强技术。从基于简单但出人意料地有效直觉构建的基础模型VTA-LDM开始，我们通过消融研究探索各种视觉编码器和辅助嵌入。通过采用强调生成质量和视频-音频同步对齐的全面评估流程，我们展示了我们的模型具有最先进的视频到音频生成能力。此外，我们提供了关于不同数据增强方法对增强生成框架整体能力的影响的重要见解。我们展示了推进从语义和时间角度生成同步音频的挑战的可能性。我们希望这些见解将成为开发更加真实和准确的音视频生成模型的奠基石。

English

Generating semantically and temporally aligned audio content in accordance with video input has become a focal point for researchers, particularly following the remarkable breakthrough in text-to-video generation. In this work, we aim to offer insights into the video-to-audio generation paradigm, focusing on three crucial aspects: vision encoders, auxiliary embeddings, and data augmentation techniques. Beginning with a foundational model VTA-LDM built on a simple yet surprisingly effective intuition, we explore various vision encoders and auxiliary embeddings through ablation studies. Employing a comprehensive evaluation pipeline that emphasizes generation quality and video-audio synchronization alignment, we demonstrate that our model exhibits state-of-the-art video-to-audio generation capabilities. Furthermore, we provide critical insights into the impact of different data augmentation methods on enhancing the generation framework's overall capacity. We showcase possibilities to advance the challenge of generating synchronized audio from semantic and temporal perspectives. We hope these insights will serve as a stepping stone toward developing more realistic and accurate audio-visual generation models.