通過通用關鍵影格提取橋接VideoQA與影片引導的智能體任務

摘要

影片理解是多模態智慧的基本能力，近年來的多模態大型語言模型（MLLMs）在影片問答（VideoQA）基準測試上展現出色表現。然而，現有基準主要評估模型能否感知淺層視覺線索，卻鮮少檢驗多模態大型語言模型是否能從影片教學中學習更深層知識或程序性技能，並將其泛化至下游的長時程代理任務。為彌補此缺口，我們提出VG-GUIBench（影片引導式圖形使用者介面基準），這是一個全新基準，旨在評估基於多模態大型語言模型的圖形使用者介面代理能否遵循影片教學完成對應的圖形使用者介面互動任務。此外，我們觀察到模型在影片問答與影片引導代理任務的表現，關鍵取決於有效的關鍵影格萃取。基於此觀察，我們提出TASKER（任務驅動與場景感知關鍵影格搜尋器），這是一種同時考量任務關聯性與場景動態的關鍵影格萃取演算法，用以識別富含資訊的影格。實驗結果顯示，TASKER在影片問答與影片引導代理任務基準上均取得顯著效能提升，分別在EgoSchema完整集與NExT-QA資料集上優於最佳基準線2.0%與1.8%。這些結果進一步凸顯通用關鍵影格萃取方法在影片理解任務上的潛力。我們的程式碼與資料可於 https://github.com/VG-GUI-TASKER/VG-GUI-TASKER 取得。

English

Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.