LiveCodeBench Pro：奧林匹亞獎牌得主如何評判大型語言模型在競技編程中的表現？
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
June 13, 2025
作者: Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, Saining Xie
cs.AI
摘要
近期有報告指出，大型語言模型（LLMs）在競技編程領域已超越頂尖人類選手。基於一組國際算法競賽獎牌得主的專業知識，我們重新審視這一主張，探討LLMs與人類專家之間的差異及其仍存的局限。我們推出了LiveCodeBench Pro，這是一個由Codeforces、ICPC和IOI題目組成的基準測試，持續更新以降低數據污染的風險。一支由奧林匹克競賽獎牌得主組成的團隊為每道題目標註算法類別，並對模型生成失敗的提交進行逐行分析。利用這些新數據和基準測試，我們發現前沿模型仍存在顯著限制：在沒有外部工具輔助的情況下，最佳模型在中等難度題目上的通過率僅為53%，而在難題上則為0%，這些領域中人類專家仍表現卓越。我們還發現，LLMs在實現密集型的問題上表現出色，但在細膩的算法推理和複雜的案例分析上卻舉步維艱，常常生成自信滿滿卻錯誤的解釋。高性能似乎主要依賴於實現的精確性和工具的增強，而非卓越的推理能力。因此，LiveCodeBench Pro不僅凸顯了與人類大師級水平之間的顯著差距，還提供了細緻的診斷，以引導未來在代碼中心化LLM推理方面的改進方向。
English
Recent reports claim that large language models (LLMs) now outperform elite
humans in competitive programming. Drawing on knowledge from a group of
medalists in international algorithmic contests, we revisit this claim,
examining how LLMs differ from human experts and where limitations still
remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from
Codeforces, ICPC, and IOI that are continuously updated to reduce the
likelihood of data contamination. A team of Olympiad medalists annotates every
problem for algorithmic categories and conducts a line-by-line analysis of
failed model-generated submissions. Using this new data and benchmark, we find
that frontier models still have significant limitations: without external
tools, the best model achieves only 53% pass@1 on medium-difficulty problems
and 0% on hard problems, domains where expert humans still excel. We also find
that LLMs succeed at implementation-heavy problems but struggle with nuanced
algorithmic reasoning and complex case analysis, often generating confidently
incorrect justifications. High performance appears largely driven by
implementation precision and tool augmentation, not superior reasoning.
LiveCodeBench Pro thus highlights the significant gap to human grandmaster
levels, while offering fine-grained diagnostics to steer future improvements in
code-centric LLM reasoning.