观察、记忆、推理：基于多模态大语言模型的人类视角视频理解

摘要

视频理解正被多模态大语言模型（MLLMs）快速重塑，研究焦点已从短视频片段转向长视频、多模态及知识密集型视频场景。这些场景要求模型在有限计算资源下处理稀疏证据、长程依赖、多模态对齐并实现可靠推理。本文以人类视角审视基于LLM的视频理解，围绕三大功能能力组织：观看、记忆与推理。该视角并非将视频任务视为孤立基准，而是为分析视频MLLM如何获取证据、保持上下文并生成基于事实的输出提供统一框架。我们提出一种形式化描述，将视频理解系统表征为感知表征、记忆状态、推理轨迹与最终预测。基于此形式化描述，我们识别出时空感知、高效长视频处理、记忆建模、流式理解及忠实推理等挑战。代表性方法按其视频MLLM系统中的角色进行组织：观看涵盖细粒度、全方位、音视频融合及高效感知；记忆包括离线与流式记忆；推理则涵盖纯文本推理与基于视频的思维。进一步考察了自我中心视角、体育、教学、医疗及叙事视频等应用领域，并覆盖了跨任务类型、监督格式、模态及能力维度的训练数据集与评估基准。最后，我们概述了可扩展、具记忆意识且基于证据的视频智能所面临的开放问题与未来方向。相关工作将持续追踪于 https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding。

English

Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.