HarnessEval-W：視覺世界評估的智能體化

摘要

一個基準不應僅提供標量分數：評估之所以可信，在於支撐該分數的推理過程。這對世界模型尤其關鍵，因為判斷一次推演的優劣，需要理解物理、因果關係及世界狀態是否正確演進。人類能自然地察覺此類違背，然而現有基準均未自動化此能力：指標以暴力計算方式產生，未留下任何可審查或驗證的推理鏈。我們提出 HarnessEval-W，這是一個智能體化的評估管線，將 LLM 生態系中的測試框架（harness）範式引入世界模型基準測試。HarnessEval-W 不以固定評分標準進行評估，而是解讀每個評估案例的脈絡，將評估問題分解為可量測的子問題，並生成專門化的子智能體，每個子智能體配備量身打造的脈絡與診斷工具，就其各自的子問題進行推理。父智能體隨後驗證所蒐集的證據，並將其彙整為最終判定。此層級化工作流程將每次評估轉化為透明的證據樹，其完整的推理鏈為結果提供依據。我們將 HarnessEval-W 應用於 18 個具代表性的世界模型，涵蓋 330 個評估案例。其判定與人類偏好高度一致，同時為每次生成的推演提供可驗證且細粒度的診斷。我們將完整管線開源作為持續運行的基準，並邀請廣大社群共同貢獻，隨著世界模型的演進，持續拓展新的技能與評估案例。

English

A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.