音频故事：利用大语言模型生成长篇叙事音频

AudioStory: Generating Long-Form Narrative Audio with Large Language Models

August 27, 2025

作者: Yuxin Guo, Teng Wang, Yuying Ge, Shijie Ma, Yixiao Ge, Wei Zou, Ying Shan

cs.AI

摘要

近期，文本到音频（TTA）生成技术在合成短音频片段方面取得了显著进展，但在处理长篇叙事音频时仍面临挑战，这类音频需要时间上的连贯性和组合推理能力。为填补这一空白，我们提出了AudioStory，一个将大型语言模型（LLMs）与TTA系统相融合的统一框架，旨在生成结构化的长篇音频叙事。AudioStory具备强大的指令跟随与推理生成能力，它利用LLMs将复杂的叙事查询分解为带有上下文提示的时间顺序子任务，从而实现场景间的平滑过渡及情感基调的一致性。AudioStory拥有两大亮点：(1)解耦的桥梁机制：AudioStory将LLM与扩散模型的协作拆分为两个专门组件，即用于事件内语义对齐的桥梁查询和用于跨事件连贯性保持的残差查询。(2)端到端训练：通过将指令理解与音频生成统一于单一端到端框架内，AudioStory无需模块化训练管道，同时增强了组件间的协同效应。此外，我们建立了AudioStory-10K基准，涵盖动画音景和自然声音叙事等多个领域。大量实验表明，AudioStory在单音频生成和叙事音频生成上均表现出色，在指令跟随能力和音频保真度上均超越了先前的TTA基线。我们的代码已发布于https://github.com/TencentARC/AudioStory。

English

Recent advances in text-to-audio (TTA) generation excel at synthesizing short audio clips but struggle with long-form narrative audio, which requires temporal coherence and compositional reasoning. To address this gap, we propose AudioStory, a unified framework that integrates large language models (LLMs) with TTA systems to generate structured, long-form audio narratives. AudioStory possesses strong instruction-following reasoning generation capabilities. It employs LLMs to decompose complex narrative queries into temporally ordered sub-tasks with contextual cues, enabling coherent scene transitions and emotional tone consistency. AudioStory has two appealing features: (1) Decoupled bridging mechanism: AudioStory disentangles LLM-diffuser collaboration into two specialized components, i.e., a bridging query for intra-event semantic alignment and a residual query for cross-event coherence preservation. (2) End-to-end training: By unifying instruction comprehension and audio generation within a single end-to-end framework, AudioStory eliminates the need for modular training pipelines while enhancing synergy between components. Furthermore, we establish a benchmark AudioStory-10K, encompassing diverse domains such as animated soundscapes and natural sound narratives. Extensive experiments show the superiority of AudioStory on both single-audio generation and narrative audio generation, surpassing prior TTA baselines in both instruction-following ability and audio fidelity. Our code is available at https://github.com/TencentARC/AudioStory