基於自迴歸擴散Transformer的串流同步空間音訊生成

摘要

实时且精确的空间音频生成对于提供沉浸式体验至关重要。然而，现有的空间音频合成技术常受限于生成质量与高推理延迟之间的权衡，且难以从多模态输入中捕捉精确的空间信息。为解决这些挑战，我们提出SwanSphere——一个面向全景视频与文本提示的高保真空间音频统一流式生成框架。SwanSphere的主要贡献如下：1) 我们引入了一种因果自回归扩散变换器架构，能够实现流式高质量空间音频生成；2) 设计了空间视频-音频对比学习策略以对齐视频编码器与声学领域，并进一步采用多目标在线直接偏好优化方案，从而赋予模型强大的空间感知能力与稳健的多模态空间音频合成性能；3) 为缓解当前空间音频数据集的匮乏，我们还开发了一套自动化标注流程以生成详细的空间描述。实验结果表明，SwanSphere在视频到空间音频及文本到空间音频的生成任务中均实现了卓越性能。演示内容可访问：https://swanaigc.github.io。

English

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.