使用预训练语言模型进行代码执行

摘要

代码执行是编程语言语义的一个基本方面，反映了代码的确切行为。然而，大多数用于代码智能的预训练模型忽略了执行跟踪，只依赖于源代码和句法结构。本文研究了预训练模型在理解和执行代码方面的表现。我们开发了一种基于变异的数据增强技术，创建了一个大规模且真实的Python数据集和代码执行任务，挑战了诸如Codex之类的现有模型。然后，我们提出了CodeExecutor，一个利用代码执行预训练和课程学习来增强其语义理解能力的Transformer模型。我们对CodeExecutor进行了代码执行评估，展示了其有希望的性能和局限性。我们还展示了它对于代码智能任务（如零样本代码到代码搜索和文本到代码生成）的潜在益处。我们的分析提供了关于预训练模型在代码执行方面学习和泛化能力的见解。

English

Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code. However, most pre-trained models for code intelligence ignore the execution trace and only rely on source code and syntactic structures. In this paper, we investigate how well pre-trained models can understand and perform code execution. We develop a mutation-based data augmentation technique to create a large-scale and realistic Python dataset and task for code execution, which challenges existing models such as Codex. We then present CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension. We evaluate CodeExecutor on code execution and show its promising performance and limitations. We also demonstrate its potential benefits for code intelligence tasks such as zero-shot code-to-code search and text-to-code generation. Our analysis provides insights into the learning and generalization abilities of pre-trained models for code execution.

使用预训练语言模型进行代码执行

Code Execution with Pre-trained Language Models

摘要

Support