修剪视觉世界建模评估的长尾

摘要

物理交互遵循长尾分布：一组常见且规律的交互主导了人类经验与视觉数据，而广泛多样、罕见且不规律的交互则未得到充分呈现。尽管当前的视觉世界模型（包括图像与视频生成模型）在现有基准测试中展现出惊人的逼真度，但它们主要聚焦于模拟常见的物理交互。这引出一个核心问题：当前的视觉世界模型是否真正内化并泛化了物理原理？在本工作中，我们提出了Tailor-Bench基准，旨在挑战世界模型模拟不规律物理交互的能力。为便于系统评估，我们设计了三种逐步挑战模型推理能力的场景模式：常规场景（Regular）反映常见的工具-任务组合；非常规场景（Unconventional）用属性兼容的替代工具替换传统工具，以测试功能泛化能力；不可能场景（Impossible）则引入违反属性的工具，以探究约束感知能力。此外，我们在统一的评估协议下设计了两种互补设定：预测生成要求在无引导下推断结果，而描述生成则指定目标结果，要求模型忠实实现。实验结果表明，物理世界建模中存在明显的长尾鸿沟：模型性能从常规场景到非常规场景再到不可能场景逐步下降，表明其在常见交互之外的泛化能力有限。失败分析进一步揭示，模型依赖于表面视觉模式：图像模型无法实现正确的状态变化，而视频模型还受限于时间不一致性。

English

Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a broad spectrum of rare and irregular interactions remains underrepresented. Although recent visual world models, including image and video generation models, achieve impressive realism on existing benchmarks, they primarily focus on simulating common physical interactions. This raises a central question: Do current visual world models internalize and generalize physical principles? In this work, we introduce Tailor-Bench, a benchmark that challenges world models to simulate irregular physical interactions. To enable systematic evaluation, we design three scenario modes that progressively challenge model reasoning: Regular scenarios reflect common tool-task pairs, Unconventional scenarios replace conventional tools with attribute-compatible substitutes to test affordance generalization, and Impossible scenarios introduce attribute-violating tools to probe constraint awareness. Additionally, we design two complementary settings under a unified evaluation protocol: predictive generation requires inferring outcomes without guidance, while descriptive generation specifies the target outcome for faithful realization. Our experimental results reveal a clear long-tail gap in physical world modeling: performance degrades from Regular to Unconventional and Impossible scenarios, indicating limited generalization beyond common interactions. Failure analysis further shows that models rely on superficial visual patterns: image models fail to realize correct state changes, while video models further suffer from temporal inconsistencies.