LiveCodeBench Pro：奥赛奖牌得主如何评判大语言模型在竞技编程中的表现？
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
June 13, 2025
作者: Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, Saining Xie
cs.AI
摘要
近期报告声称，大型语言模型（LLMs）在竞技编程领域已超越人类精英。基于一组国际算法竞赛奖牌得主的专业知识，我们重新审视了这一论断，探讨了LLMs与人类专家的差异及其仍存的局限。我们推出了LiveCodeBench Pro，这是一个由Codeforces、ICPC和IOI题目构成的基准测试集，持续更新以减少数据污染的可能性。一支由奥赛奖牌得主组成的团队为每道题目标注了算法类别，并对模型生成的失败提交进行了逐行分析。利用这一新数据和基准测试，我们发现前沿模型仍存在显著局限：在没有外部工具辅助的情况下，最佳模型在中等难度题目上的通过率仅为53%，而在高难度题目上则为0%，这些领域人类专家依然表现出色。我们还发现，LLMs在实现密集型的题目上表现良好，但在细微的算法推理和复杂案例分析上则显得吃力，常常生成自信但错误的解释。高性能似乎主要源于实现的精确性和工具的增强，而非卓越的推理能力。因此，LiveCodeBench Pro不仅凸显了与人类大师级水平之间的显著差距，还提供了细粒度的诊断，以指导未来代码中心化LLM推理能力的提升。
English
Recent reports claim that large language models (LLMs) now outperform elite
humans in competitive programming. Drawing on knowledge from a group of
medalists in international algorithmic contests, we revisit this claim,
examining how LLMs differ from human experts and where limitations still
remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from
Codeforces, ICPC, and IOI that are continuously updated to reduce the
likelihood of data contamination. A team of Olympiad medalists annotates every
problem for algorithmic categories and conducts a line-by-line analysis of
failed model-generated submissions. Using this new data and benchmark, we find
that frontier models still have significant limitations: without external
tools, the best model achieves only 53% pass@1 on medium-difficulty problems
and 0% on hard problems, domains where expert humans still excel. We also find
that LLMs succeed at implementation-heavy problems but struggle with nuanced
algorithmic reasoning and complex case analysis, often generating confidently
incorrect justifications. High performance appears largely driven by
implementation precision and tool augmentation, not superior reasoning.
LiveCodeBench Pro thus highlights the significant gap to human grandmaster
levels, while offering fine-grained diagnostics to steer future improvements in
code-centric LLM reasoning.