智能体化超长视频理解

摘要

随着智能眼镜等全天候可穿戴设备的普及，始终在线的人工智能个人助手对情境理解提出了更高要求——这种理解需突破短暂孤立事件的局限，涵盖连续纵向的第一人称视角视频流。实现这一愿景需要长周期视频理解技术的进步，即系统必须能解读并回溯跨越数日甚至数周的视听信息。现有方法（包括大语言模型和检索增强生成技术）受限于上下文窗口的约束，难以对超长视频流进行组合式多跳推理。本研究通过EGAgent这一以实体场景图为核心的增强型智能体框架应对上述挑战：该图结构可动态表征人物、场景、物体及其随时间演化的关联关系。我们的系统为规划智能体配备了结构化图检索推理工具及混合视听搜索能力，从而实现细粒度、跨模态且时序连贯的推理。在EgoLifeQA和Video-MME（Long）数据集上的实验表明，本方法在复杂长周期视频理解任务中达到EgoLifeQA最高性能（57.5%），并在Video-MME（Long）上取得具有竞争力的表现（74.1%）。

English

The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks.

智能体化超长视频理解

Agentic Very Long Video Understanding

摘要

Support