Vinci2: 在连续自我中心视频中提供主动辅助

摘要

智能助手何时应主动发言？连续的第一人称视频提供了丰富且不断演化的上下文，使得一种新型的辅助方式成为可能：这种辅助是主动的，而不仅仅是反应式的。然而，现有方法要么被动等待用户提问，要么将每个检测到的事件都视为需要响应，而不考虑用户的历史记录、当前活动，或者这种辅助是否真的受欢迎。我们将主动辅助重新定义为一个依赖于上下文的决策问题：智能体不仅要感知正在发生的事情，还要基于累积的时间上下文进行推理，以决定何时以及是否要介入。为此，我们提出了Vinci2，一个主动的第一人称辅助系统，它将设备端助手Vinci从被动响应升级为主动交互。在评估方面，我们推出了EgoServe，这是首个用于连续第一人称视频中主动辅助的大规模基准。EgoServe包含3000多个服务实例，按照4个时间记忆范围（从即时安全警报到长期习惯辅导）组织，涵盖10个服务类别。在建模方面，我们提出了EgoMemo，这是一种无需训练、基于记忆增强的智能体，它维护三种互补的记忆表示：多尺度时间摘要、语义知识图谱和视觉嵌入档案。在每个时间步，EgoMemo通过检索增强推理来判断是否需要提供辅助，并在需要时生成基于上下文的响应。实验表明，EgoMemo在EgoServe上建立了强大的基线，同时在现有的第一人称基准上具有竞争力。我们的基准和代码公开在 https://sitonggong.github.io/EgoServe-page/{Vinci2}。

English

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at https://sitonggong.github.io/EgoServe-page/{Vinci2}.