报告卡：利用自然语言摘要对语言模型进行定性评估

摘要

大型语言模型（LLMs）的快速发展和动态特性使得传统的量化基准难以准确评估其能力。我们提出了一种名为“报告卡”的方法，即以人类可理解的自然语言形式，针对特定技能或主题总结模型行为。我们构建了一个基于三个标准的框架来评估报告卡：特异性（区分不同模型的能力）、忠实性（准确反映模型能力）和可解释性（对人类而言的清晰度和相关性）。此外，我们提出了一种无需人工监督的迭代算法来生成报告卡，并通过消融实验探讨了不同设计方案的有效性。通过对流行LLMs的实验，我们证明报告卡能够提供超越传统基准的洞察，有助于满足对LLMs进行更可解释和全面评估的需求。

English

The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

报告卡：利用自然语言摘要对语言模型进行定性评估

Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

摘要

Support