VGI-Bench：探测视频生成模型的视觉智能

摘要

近期研究表明，视频生成模型可以通过生成的帧展现出某种形式的零样本视觉推理能力。然而，可靠的评估仍然具有挑战性：基准测试应采用与当前视频模型视觉先验一致的输入，需要有效的演化过程而不仅仅是合理的最终状态，并且需要校准任务难度，使其既具有挑战性又部分可行。为此，我们提出了VGI-bench，包含27个任务和810个实例，通过任务领域和技能标签的两级分类体系来组织，用于对视频生成模型的视觉推理能力进行细粒度评估。我们的评估表明，当前的生成系统可以解决一部分基于视觉的推理任务，但远未达到可靠的水平，即使是最强的模型Seedance 2.0，在我们的评估标准下也仅达到51.0%。我们进一步分析了输出失败模式、输入条件敏感性、合成微调的性能迁移边界，以及内部去噪视角所揭示的有限自我修正能力——后续步骤主要是在优化早期假设，而非纠正推理错误。我们希望VGI-bench能够推动下一代视频生成模型的发展。网站：https://hexuan21.github.io/VGI-Bench/

English

Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/

VGI-Bench：探测视频生成模型的视觉智能

VGI-Bench: Probing Visual Intelligence in Video Generation Models

摘要

Support