代碼語言模型研究綜述

摘要

在這份研究中，我們系統性地回顧了最近在代碼處理方面與語言模型相關的進展，涵蓋了50多種模型、30多個評估任務和500篇相關作品。我們將代碼處理模型分為一般語言模型（如GPT家族）和專門在代碼上預訓練的特殊模型，通常具有定制目標。我們討論了這些模型之間的關係和差異，並突出了代碼建模從統計模型和RNN到預訓練Transformer和LLM的歷史轉變，這正是NLP所採取的相同路徑。我們還討論了代碼特定功能，如AST、CFG和單元測試，以及它們在訓練代碼語言模型中的應用，並確定了這一領域中的關鍵挑戰和潛在未來方向。我們將這份調查保持開放並更新在GitHub存儲庫上，網址為https://github.com/codefuse-ai/Awesome-Code-LLM。

English

In this work we systematically review the recent advancements in code processing with language models, covering 50+ models, 30+ evaluation tasks, and 500 related works. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and LLMs, which is exactly the same course that had been taken by NLP. We also discuss code-specific features such as AST, CFG, and unit tests, along with their application in training code language models, and identify key challenges and potential future directions in this domain. We keep the survey open and updated on github repository at https://github.com/codefuse-ai/Awesome-Code-LLM.

代碼語言模型研究綜述

A Survey on Language Models for Code

摘要

Support