挖掘模型仓库中的隐藏瑰宝

摘要

公共代码库托管着数百万个精调模型，但社区使用量仍不成比例地集中在少数基础模型上。本研究旨在探究这种集中现象究竟反映了有效的市场选择，还是存在系统性忽视优质模型的情况。通过对2000多个模型进行广泛评估，我们发现"隐藏瑰宝"现象普遍存在——这些冷门的精调模型显著优于热门模型。值得注意的是，在Llama-3.1-8B模型系列中，我们发现了下载量极低的检查点，其数学推理能力从83.2%提升至96.0%，且未增加推理成本。然而，通过对每个上传模型进行穷举评估来发现优质模型在计算上是不可行的。为此，我们将模型发现问题建模为多臂老虎机问题，通过采用共享查询集和激进淘汰机制，对序贯二分搜索算法进行加速。我们的方法仅需对每个候选模型进行50次查询即可定位最优模型，将发现效率提升超过50倍。

English

Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.

挖掘模型仓库中的隐藏瑰宝

Discovering Hidden Gems in Model Repositories

摘要

Support