Motion-Omni：用於口語對話的端到端聯合語音與全身動作生成

摘要

一個進行對話的虛擬化身必須決定要說甚麼，以及在說話的同時如何動作；然而，這些能力分屬不同的模型家族：語音對話模型生成語音但不含動作，而語音伴隨動作模型只能從輸入的音訊生成動作。標準的解決方式是級聯（cascade）架構：先產生語音回應，再對完成後的音訊執行動作模型；這需要第二次完整的推論過程，並排除了兩者之間進行聯合最佳化的可能。我們提出 Motion-Omni，這是一個端到端框架，其中的語音對話模型能原生輸出明確的面部表情，以及手部、上半身與下半身的動作，且這些動作直接由產生語音的隱藏狀態生成。在此架構中，聯合訓練並非可選項：若語音路徑被凍結，動作仍會與音訊不對齊；唯有在語音與動作兩個目標下共同調適 LLM、語音生成器與動作生成器，才能在保留語音對話能力的同時恢復對齊。監督訊號來自一個可擴展、與模型無關的流程，該流程以可替換的動作教師模型，為語音一致的語音回應建立偽標籤，產出 422,856 對依品質排序的配對（共 1,402 小時）。我們進一步釋出 SwDA-500，以及據我們所知首個針對隨機開放式全身語音對話的公開評估協議；該協議在跨動作系統間匹配音訊，同時統整了渲染、自動指標、人工評估與延遲量測。以 Qwen2.5-7B-Instruct 為骨幹實例化後，Motion-Omni-Q7 在無參考動作指標上與使用相同音訊的教師級聯差距在 2% 以內，但回應速度快 5.4 倍（RTF=0.78，快於即時）；在節拍相關性與多樣性上超越所有非教師級聯，並達到 2.62% 的字錯誤率，為所比較之全模態系統中最低。

English

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.