FACTS权威排行榜：大型语言模型事实准确性综合基准评测

摘要

我们正式推出FACTS评估体系——一套在线排行榜系统及配套基准测试，旨在全面评估语言模型在不同场景下生成事实准确文本的能力。该体系通过聚合模型在四个独立子榜单上的表现提供整体事实性度量：(1) FACTS多模态榜单，衡量基于图像提问的回复事实性；(2) FACTS参数化榜单，通过闭卷事实性问题评估模型从内部参数提取世界知识的能力；(3) FACTS搜索榜单，评估信息检索场景下模型使用搜索API时的事实准确性；(4) FACTS文本锚定榜单（v2版），评估长文本回复是否基于给定文档，其判定模型得到显著优化。各子榜单均采用自动化判定模型对回答进行评分，最终体系得分为四项分值的平均值，以此实现对模型整体事实性的稳健均衡评估。FACTS评估体系将保持动态更新，同时包含公开与私有数据分区，在保障体系完整性的同时支持外部参与。访问地址：https://www.kaggle.com/benchmarks/google/facts。

English

We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .

FACTS权威排行榜：大型语言模型事实准确性综合基准评测

The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality

摘要

Support