休謨：衡量文本嵌入任務中人機表現差距

摘要

比较人类与模型的表现，为理解嵌入模型的优势与局限提供了宝贵的视角，揭示了它们在捕捉意义与细微差别方面的成功与失败之处。然而，此类比较鲜有进行，因为人类在嵌入任务上的表现难以量化。为填补这一空白，我们引入了HUME：文本嵌入的人类评估框架。尽管如MTEB等框架提供了广泛的模型评估，但它们缺乏对人类表现的可靠估计，限制了模型得分的可解释性。我们测量了人类在16个MTEB数据集上的表现，这些数据集涵盖了重排序、分类、聚类及跨语言多样性高、低资源语言的语义文本相似性任务。人类平均表现达到77.6%，而最佳嵌入模型为80.1%，尽管差异显著：模型在某些数据集上接近天花板表现，而在其他数据集上则表现挣扎，暗示了数据集问题并揭示了低资源语言中的不足。我们提供了人类表现的基准、对任务难度模式的洞察，以及一个可扩展的评估框架，该框架不仅使模型解释更具意义，还指导了模型与基准的发展。我们的代码、数据集及排行榜公开于https://github.com/embeddings-benchmark/mteb。

English

Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, although variation is substantial: models reach near-ceiling performance on some datasets while struggling on others, suggesting dataset issues and revealing shortcomings in low-resource languages. We provide human performance baselines, insight into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of the model and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.

休謨：衡量文本嵌入任務中人機表現差距

HUME: Measuring the Human-Model Performance Gap in Text Embedding Task

摘要

Support