VGI-Bench：探測視頻生成模型中的視覺智能

摘要

近期研究表明，影片生成模型可以透過生成的幀展現出某種形式的零樣本視覺推理。然而，可靠的評估仍具挑戰性：基準應採用與當前影片模型視覺先驗一致的輸入，要求有效的演化過程而不僅是合理的最終狀態，並校準任務難度，使其既有挑戰性又部分可行。為此，我們提出了 VGI-bench，包含 27 個任務和 810 個實例，按任務領域與技能標籤的兩級分類法組織，用於對影片生成模型的視覺推理能力進行細粒度評估。我們的評估顯示，當前的生成系統能解決一部分視覺基礎推理任務，但遠未達到可靠程度，即使是最強的模型 Seedance 2.0，在我們的評估標準下也僅達到 51.0%。我們的分析進一步探討了輸出失敗模式、輸入條件敏感性、合成微調的性能遷移邊界，以及內部去噪視角所揭示的有限自校正能力：後續步驟主要是在完善早期假設，而非糾正推理錯誤。我們希望 VGI-bench 能有助於推動下一代影片生成模型的發展。網站：https://hexuan21.github.io/VGI-Bench/

English

Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/

VGI-Bench：探測視頻生成模型中的視覺智能

VGI-Bench: Probing Visual Intelligence in Video Generation Models

摘要

Support