CodeEditorBench：評估大型語言模型的程式碼編輯能力

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

April 4, 2024

作者: Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, Jie Fu

cs.AI

摘要

大型語言模型（LLMs）用於程式碼的應用正在迅速演進，程式碼編輯已成為一項關鍵能力。我們引入了 CodeEditorBench，這是一個旨在嚴謹評估LLMs在程式碼編輯任務中表現的評估框架，包括除錯、翻譯、修改和需求切換等任務。與現有僅關注程式碼生成的基準不同，CodeEditorBench強調現實世界情境和軟體開發的實際面向。我們從五個來源中精心挑選了多樣的編碼挑戰和情境，涵蓋各種程式語言、複雜度水平和編輯任務。對19個LLMs的評估顯示，封閉源模型（尤其是Gemini-Ultra和GPT-4）在CodeEditorBench中優於開源模型，突顯了基於問題類型和提示敏感性的模型表現差異。CodeEditorBench的目標是通過提供一個堅固的平台來評估程式碼編輯能力，以催生LLMs的進步。我們將釋出所有提示和資料集，以便社群擴展資料集並對新興LLMs進行基準測試。通過引入CodeEditorBench，我們促進了LLMs在程式碼編輯方面的進步，並為研究人員和從業者提供了寶貴的資源。

English

Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.