報告卡：使用自然語言摘要對語言模型進行定性評估

摘要

大型語言模型（LLMs）的快速發展與動態特性，使得傳統量化基準難以準確評估其能力。我們提出「能力報告卡」概念——針對特定技能或主題、以人類可解讀的自然語言呈現模型行為摘要。我們建立了一套評估框架，基於三項標準衡量報告卡效能：區分度（區分不同模型的能力）、忠實度（準確反映模型能力）及可解讀性（對人類而言的清晰度與相關性）。同時提出一種無需人工監督即可生成報告卡的迭代演算法，並透過消融實驗驗證各設計要素的效用。透過對主流LLMs的實驗驗證，我們證明能力報告卡能提供超越傳統基準的洞察，有助於實現更可解讀、更全面的LLM評估需求。

English

The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

報告卡：使用自然語言摘要對語言模型進行定性評估

Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

摘要

Support