任務我任何事

摘要

大型多模態語言模型（MLM）的基準現在用於同時評估模型的一般能力，而不僅僅是評估特定能力。因此，當開發人員想要確定應該為其應用程序選擇哪些模型時，他們會被眾多基準所淹沒，並且對哪個基準的結果最能反映其特定用例感到不確定。本文介紹了一個名為「Task-Me-Anything」的基準生成引擎，它可以生成符合用戶需求的基準。Task-Me-Anything保持了一個可擴展的視覺資產分類法，可以以程式方式生成大量任務實例。此外，它可以在計算預算內高效地以演算法方式回答用戶關於MLM性能的查詢。它包含了113K張圖像、10K個視頻、2K個3D物體資產、365多個物體類別、655個屬性和335個關係。它可以生成750M個圖像/視頻問答對，重點評估MLM的感知能力。Task-Me-Anything揭示了一些關鍵見解：開源MLM在物體和屬性識別方面表現出色，但缺乏空間和時間理解；每個模型都有獨特的優勢和劣勢；通常較大的模型表現更好，但也存在例外；而GPT4o在識別旋轉/移動物體和區分顏色方面存在挑戰。

English

Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.

任務我任何事

Task Me Anything

摘要

Summary

Support

Support