LatentOmni：通过统一视听潜在推理重新思考全模态理解

摘要

聯合音視頻推理對於全方位理解至關重要，然而當前多模態大型語言模型在需要從兩種模態中提取細粒度證據進行推理時，仍面臨困難。一個核心限制在於，基於明確文本的思維鏈會將連續的音視頻信號壓縮為離散標記，削弱時間定位能力，並使中間推理偏向語言先驗。我們認為，統一的潛空間是此類推理更佳的媒介，因為它保留了密集的感官信息，同時仍與自回歸生成兼容。基於此見解，我們提出LatentOmni，這是一個跨模態推理框架，將文本推理與音視頻潛狀態交錯結合。LatentOmni引入特徵層級監督，使潛推理狀態與任務相關的感官特徵對齊，並使用Omni-Sync位置嵌入來維持潛在音頻與視頻狀態之間的時間一致性。我們進一步建構了LatentOmni-Instruct-35K數據集，其中包含用於監督潛空間推理的音視頻交錯推理軌跡。跨越多個音視頻推理基準的全面評估顯示，LatentOmni在評估的開源模型中表現最佳，並持續優於明確文本思維鏈基線，驗證了潛空間聯合推理是邁向更強大全方位理解的有效途徑。

English

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose LatentOmni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.