HAKARI-Bench：一個在統一條件下比較檢索架構與效率設定的輕量級基準測試

摘要

隨著檢索增強生成與語義搜索的迅速普及，選擇合適的嵌入與檢索配置變得日益困難。大型檢索基準雖然全面，但在開發過程中重新運行的負擔過重，並且缺乏在相同條件下比較多個模型的生產環境設置（如維度縮減、量化、重排序）的基礎設施。我們提出HAKARI-Bench，這是一個輕量級基準，將現有檢索套件重構為小型資料集（Nano集）：涵蓋35個基準、551項任務，橫跨43種語言，採用統一格式，支援在相同條件下、無關模型地比較五類檢索家族（BM25、密集檢索、稀疏檢索、後期交互、重排序器）及其效率變體。在55個模型上，其整體排名與官方MTEB檢索v2、MMTEB v2檢索以及英文BEIR（完整版）的斯皮爾曼相關係數均高於0.97。HAKARI-Bench並非取代完整評估；而是實現快速模型選擇、回歸檢測以及識別品質與效率的帕累托前沿。程式碼、資料與排行榜均以MIT許可證釋出。

English

With the rapid spread of retrieval-augmented generation and semantic search, choosing the right embedding and retrieval configuration is increasingly hard. Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings--dimensionality reduction, quantization, reranking--across many models under identical conditions. We present HAKARI-Bench, a lightweight benchmark that reconstructs existing retrieval suites into small datasets (Nano-sets): 35 benchmarks and 551 tasks across 43 languages in a unified format, enabling same-condition, model-agnostic comparison of five retrieval families (BM25, dense, sparse, late interaction, rerankers) and their efficiency variants. Across 55 models, its overall ranking reproduces the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) at Spearman >0.97. HAKARI-Bench does not replace full evaluation; it enables rapid model selection, regression detection, and reading the quality-efficiency Pareto frontier. Code, data, and leaderboard are released under the MIT license.