AuK 技術報告：一個用於語音生成與編輯的開源基礎模型

摘要

我們提出 AuK，一個開放原始碼基礎模型，其透過自然語言指令與音訊情境的共同介面，統一語音生成與編輯。為支援如此廣泛的能力集，我們建構約 30.3 億筆指令–音訊實例，以及橫跨五個任務家族、共 195 萬小時的有效監督：語音生成、內容編輯、增強與分離、副語言編輯與聲學編輯。AuK 結合用於語意條件化的多模態大型語言模型、在語音、一般音訊與音樂上聯合訓練以進行聲學條件化的 VAE，以及混合式整流流 Transformer；後者先執行雙流 MMDiT 區塊，再執行統一的單流 DiT 區塊以進行生成。訓練始於僅生成暖身，接著進行聯合生成–編輯預訓練。我們隨後採用互補的後訓練策略：針對開放式編輯的人類回饋偏好最佳化，以及針對語音生成的基於獎勵之強化學習。為降低推論成本，我們進一步以一致性初始化與任務路由的解耦 DMD 蒸餾該模型。所得 AuK-Flash 可進行 4 步推論，且不依賴無分類器引導，並在匹配條件下相較完整模型達到 4.5 倍的實際耗時加速。實驗顯示，其在零樣本與指令控制的語音生成，以及通用指令引導編輯上具領先效能，同時在訊號層級修復任務上保持競爭力。我們釋出原始碼與模型權重，以支援可重現性與進一步研究。

English

We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.