ThinkSound：多模态大语言模型中的思维链推理在音频生成与编辑中的应用

ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing

June 26, 2025

作者: Huadai Liu, Jialei Wang, Kaicheng Luo, Wen Wang, Qian Chen, Zhou Zhao, Wei Xue

cs.AI

摘要

尽管端到端的视频到音频生成技术已取得显著进步，但要生成高保真音频，真实捕捉视觉内容的细微差别仍具挑战。如同创意产业中的专业人士，此类生成需要对视觉动态、声学环境及时间关系等要素进行复杂推理。我们提出了ThinkSound，一个创新框架，它利用思维链（CoT）推理实现视频音频的逐步交互式生成与编辑。我们的方法将过程分解为三个互补阶段：基础拟音生成，创建语义连贯的声景；通过精确用户交互进行的以对象为中心的交互式精炼；以及由自然语言指令引导的定向编辑。每一阶段，多模态大语言模型生成上下文对齐的CoT推理，指导统一的音频基础模型。此外，我们引入了AudioCoT，一个包含结构化推理注释的综合数据集，建立了视觉内容、文本描述与声音合成之间的联系。实验表明，ThinkSound在视频到音频生成方面，无论是音频指标还是CoT指标，均达到了业界领先水平，并在分布外电影音频基准测试中表现卓越。演示页面请访问https://ThinkSound-Project.github.io。

English

While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, such generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, a multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce AudioCoT, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis. Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics and excels in out-of-distribution Movie Gen Audio benchmark. The demo page is available at https://ThinkSound-Project.github.io.