LatentOmni：通过统一视听潜在推理重新思考全模态理解

摘要

联合音视频推理对于全模态理解至关重要，然而当前的多模态大语言模型在需要从两种模态中获取细粒度证据进行推理时仍面临挑战。其核心局限性在于：基于显式文本的思维链（CoT）将连续的音视频信号压缩为离散标记，削弱了时间定位能力，并使中间推理过程偏向语言先验。我们认为统一潜在空间是更优的推理媒介，因为它既能保留密集的感知信息，又能兼容自回归生成。基于这一见解，我们提出LatentOmni——一种跨模态推理框架，它将文本推理与音视频潜在状态交错融合。LatentOmni引入特征级监督，使潜在推理状态与任务相关的感知特征对齐，并利用全模态同步位置嵌入（OSPE）保持潜在音频和视频状态的时间一致性。我们进一步构建了LatentOmni-Instruct-35K数据集，包含音视频交错推理轨迹，用于监督潜在空间推理。在多个音视频推理基准上的全面评估表明，LatentOmni在评估的开源模型中取得了最佳性能，并持续优于显式文本思维链基线，证实了潜在空间联合推理是实现更强全模态理解的一条有前景的路径。

English

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose LatentOmni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.