基於凍結視覺語言模型的免訓練語音中心全模態理解

摘要

視聽理解仍是一項挑戰，因為模型必須同時解讀語音內容、視覺事件及其時間關聯。現有的全模態模型通常引入專用音訊編碼器，並依賴昂貴的音訊-視訊-文字訓練，使全模態能力與特定 VLM 骨幹緊密耦合，並可能削弱其既有的視覺與推理能力。這帶出三個問題：是否每個新 VLM 都需要原生全模態訓練；能否在不更動原始骨幹的情況下加入以語音為核心的全模態能力；以及哪些情境仍必須使用更豐富的聲學表徵。我們提出 Training-Free Omni（TFO），這是一個即插即用框架，能在不修改架構或重新進行多模態對齊的條件下，將凍結的 VLM 轉換為以語音為核心的全模態模型。TFO 使用 Whisper 提取經置信度過濾且帶時間戳記的轉錄文本，並將其經由 VLM 既有的語言介面路由傳遞，同時保持其視覺路徑不變。在涵蓋 56 項基準與 21 種語言、與原生全模態模型的配對比較中，TFO 在視聽理解上表現相當，在所有五種模型設定下均提升平均純音訊效能，並在跨語言語音上獲得顯著增益。相較於對應的原生全模態檢查點，凍結 VLM 也通常保留更強的影像／視訊理解、視覺定位、程式編寫、數學推理及醫學問答能力。這些結果顯示，出色的以語音為核心的全模態理解往往可以透過模組化的音訊轉語言路由來達成，而非依賴昂貴且限定特定骨幹的訓練。

English

Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.