SemComp-Bench：视频生成中的语义任务完成基准测试

摘要

我们提出语义任务完成视频生成（Semantic Task Completion Video Generation），这是一种面向结果的视频生成任务。在该设定下，任务成功既要求达成预期结果，也要求满足语义对齐。语义对齐刻画了参考图像与生成结果之间与任务相关的高层语义对应关系。评估聚焦于生成结果，既不需要呈现完整中间任务步骤序列，也不要求与参考图像保持传统意义上的外观一致性。为支持系统化评估，我们构建了 SemComp-Data，一个覆盖六个领域的评估数据集。每个样本包含一张参考图像、一条详细指令、一条简洁指令以及一段以结果为中心的视频片段。可扩展的四阶段筛选流水线将原始视频转换为标准化的 SemComp-Data 样本。我们进一步提出 SemComp-Bench，一种使用视觉语言模型（VLM）回答结构化二值问题的评估协议。SemComp-Bench 分别报告用于衡量结果达成度与生成可靠性的 OA 分数和 GR 分数。在代表性视频生成模型上的实验表明，在达成预期结果的同时保持参考图像中与任务相关的语义对齐，仍然具有挑战性。

English

We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.