流程图解：揭示视频大语言模型中信息的隐秘路径

摘要

视频大语言模型（VideoLLMs）将视觉语言模型的能力拓展至时空输入领域，实现了视频问答等任务。尽管VideoLLMs近期取得显著进展，但其内部关于视频与文本信息的提取和传递机制仍待深入探索。本研究采用机理可解释性技术，系统剖析了VideoLLMs的内部信息流。我们在多样化视频问答任务中发现一致规律：（1）时序推理始于早期至中层网络中的跨帧交互；（2）随后在中层网络实现渐进式视频-语言融合，该过程由视频表征与含时序概念的语言嵌入之间的对齐实现；（3）完成融合后，模型在中层至深层网络具备生成正确答案的能力；（4）基于此发现，我们证明VideoLLMs可通过选择有效信息路径（如LLaVA-NeXT-7B-Video-FT模型可削减58%注意力边）保持视频问答性能。这些发现揭示了VideoLLMs进行时序推理的内在机制，为提升模型可解释性与下游泛化能力提供了实践指导。项目页面及源代码详见https://map-the-flow.github.io。

English

Video Large Language Models (VideoLLMs) extend the capabilities of vision-language models to spatiotemporal inputs, enabling tasks such as video question answering (VideoQA). Despite recent advances in VideoLLMs, their internal mechanisms on where and how they extract and propagate video and textual information remain less explored. In this study, we investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques. Our analysis reveals consistent patterns across diverse VideoQA tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame interactions in early-to-middle layers, (2) followed by progressive video-language integration in middle layers. This is facilitated by alignment between video representations and linguistic embeddings containing temporal concepts. (3) Upon completion of this integration, the model is ready to generate correct answers in middle-to-late layers. (4) Based on our analysis, we show that VideoLLMs can retain their VideoQA performance by selecting these effective information pathways while suppressing a substantial amount of attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a blueprint on how VideoLLMs perform temporal reasoning and offer practical insights for improving model interpretability and downstream generalization. Our project page with the source code is available at https://map-the-flow.github.io

流程图解：揭示视频大语言模型中信息的隐秘路径

Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs

摘要

Support