阿爾法卓越基準

Alpha Excel Benchmark

May 7, 2025

作者: David Noever, Forrest McKee

cs.AI

摘要

本研究提出了一种新颖的基准测试方法，通过源自金融建模世界杯（FMWC）Excel竞赛的挑战来评估大型语言模型（LLMs）。我们介绍了一种将现有的113项FMWC挑战转化为可编程评估的JSON格式的方法，并利用这一数据集对多个领先的LLMs进行了性能比较。我们的研究结果显示，在不同挑战类别中，模型表现存在显著差异，特别是在模式识别任务上展现出特定优势，但在复杂数值推理方面则显得力不从心。该基准测试为评估LLMs在现实商业导向任务中的能力提供了一个标准化框架，而非局限于抽象的学术问题。本研究通过确立微软Excel每日用户——约15亿人——的熟练度作为一项有意义的评估指标，弥合了学术AI基准测试与实际商业应用之间的鸿沟，为不断发展的AI基准测试领域做出了贡献。

English

This study presents a novel benchmark for evaluating Large Language Models (LLMs) using challenges derived from the Financial Modeling World Cup (FMWC) Excel competitions. We introduce a methodology for converting 113 existing FMWC challenges into programmatically evaluable JSON formats and use this dataset to compare the performance of several leading LLMs. Our findings demonstrate significant variations in performance across different challenge categories, with models showing specific strengths in pattern recognition tasks but struggling with complex numerical reasoning. The benchmark provides a standardized framework for assessing LLM capabilities in realistic business-oriented tasks rather than abstract academic problems. This research contributes to the growing field of AI benchmarking by establishing proficiency among the 1.5 billion people who daily use Microsoft Excel as a meaningful evaluation metric that bridges the gap between academic AI benchmarks and practical business applications.