HarnessEval-W：视觉世界评估的代理化

摘要

一个基准评测应当提供的不只是单一分数：使评估值得信赖的是支撑该分数的推理过程。这一点对于世界模型尤为关键，因为评判一段生成轨迹需要理解物理规律、因果关系以及世界状态是否正确演化。人类能够自然地察觉此类违反之处，然而现有基准评测中没有任何一个能够自动化这种能力：评测指标以暴力计算方式生成，缺乏可供检查或验证的推理链。我们提出HarnessEval-W，一个智能体化的评估流水线，将LLM生态系统中的“harness”范式引入世界模型基准评测。与使用固定评分规则不同，HarnessEval-W会解读每个评估案例的上下文，将评估问题分解为可测量的子问题，并生成专门的子智能体，每个子智能体配备定制化上下文和诊断工具，针对自身子问题进行推理。随后，父智能体验证收集到的证据，并将其总结为最终判定。这一分层工作流将每次评估转化为一棵透明的证据树，其完整推理链为结果提供依据。我们基于330个评估案例，对18个具有代表性的世界模型应用了HarnessEval-W。其判断与人类偏好高度一致，同时为每条生成的轨迹提供可验证且细粒度的诊断。我们将完整流水线作为动态基准评测开源，并邀请广大社区在其发展过程中贡献新的技能与评估案例，以推动世界模型不断演进。

English

A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.