修剪視覺世界模型評估的長尾

摘要

物理交互遵循長尾分佈：一組常見且規律的交互主導了人類經驗與視覺數據，而廣泛的罕見及不規律交互則呈現不足。儘管近期包括影像生成與影片生成模型在內的視覺世界模型，在現有基準測試上達到了令人矚目的逼真度，但它們主要專注於模擬常見的物理交互。這引發了一個核心問題：當前的視覺世界模型是否內化並泛化了物理原理？在本研究中，我們提出了Tailor-Bench，這是一個挑戰世界模型模擬不規律物理交互的基準測試。為實現系統性評估，我們設計了三種逐步挑戰模型推理能力的情境模式：常規情境反映常見的工具-任務配對；非常規情境以屬性相容的替代品取代傳統工具，測試功能泛化能力；不可能情境則引入違反屬性的工具，探查模型對約束條件的意識。此外，我們在統一評估協議下設計了兩種互補設定：預測生成要求模型在無引導的情況下推斷結果，而描述生成則為模型指定目標結果，以實現忠實再現。實驗結果顯示，物理世界建模中存在明顯的長尾差距：模型表現從常規情境退化至非常規與不可能情境，顯示其在常見交互之外的泛化能力有限。失敗分析進一步表明，模型依賴於表面的視覺模式：影像模型未能實現正確的狀態變化，而影片模型則進一步受困於時間不一致性。

English

Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a broad spectrum of rare and irregular interactions remains underrepresented. Although recent visual world models, including image and video generation models, achieve impressive realism on existing benchmarks, they primarily focus on simulating common physical interactions. This raises a central question: Do current visual world models internalize and generalize physical principles? In this work, we introduce Tailor-Bench, a benchmark that challenges world models to simulate irregular physical interactions. To enable systematic evaluation, we design three scenario modes that progressively challenge model reasoning: Regular scenarios reflect common tool-task pairs, Unconventional scenarios replace conventional tools with attribute-compatible substitutes to test affordance generalization, and Impossible scenarios introduce attribute-violating tools to probe constraint awareness. Additionally, we design two complementary settings under a unified evaluation protocol: predictive generation requires inferring outcomes without guidance, while descriptive generation specifies the target outcome for faithful realization. Our experimental results reveal a clear long-tail gap in physical world modeling: performance degrades from Regular to Unconventional and Impossible scenarios, indicating limited generalization beyond common interactions. Failure analysis further shows that models rely on superficial visual patterns: image models fail to realize correct state changes, while video models further suffer from temporal inconsistencies.