通过观看电影学习音频高亮技术

摘要

近年来，视频内容的创作与消费显著增长。打造引人入胜的内容，需精心策划视觉与音频元素。尽管通过最佳视角选择或后期编辑等技术进行的视觉线索策划一直是媒体制作的核心，但其自然对应物——音频，却未经历同等程度的发展。这常导致视觉与听觉显著性之间的脱节。为弥合这一差距，我们引入了一项新颖任务：视觉引导的音频突出，旨在根据伴随视频的指引，对音频进行转换，以提供恰当的突出效果，最终营造更为和谐的视听体验。我们提出了一种灵活的、基于Transformer的多模态框架来解决此任务。为训练我们的模型，我们还引入了一个新数据集——混音数据集，该数据集利用电影中精细的音频与视频制作，提供了一种形式的免费监督。我们开发了一种伪数据生成流程，通过分离、调整和重新混音的三步过程，模拟现实世界中混音不佳的场景。我们的方法在定量与主观评估中均持续超越多个基线模型。我们还系统研究了不同类型上下文引导的影响及数据集的难度级别。我们的项目页面在此：https://wikichao.github.io/VisAH/。

English

Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.

通过观看电影学习音频高亮技术

Learning to Highlight Audio by Watching Movies

摘要

Summary

Support

Support