CodeTF：一站式Transformer程式庫，用於最先進的程式碼LLM。

CodeTF: One-stop Transformer Library for State-of-the-art Code LLM

May 31, 2023

作者: Nghi D. Q. Bui, Hung Le, Yue Wang, Junnan Li, Akhilesh Deepak Gotmare, Steven C. H. Hoi

cs.AI

摘要

程式碼智能在改變現代軟體工程中扮演著關鍵角色。最近，基於深度學習的模型，特別是基於 Transformer 的大型語言模型（LLMs），通過利用龐大的開源程式碼數據和程式語言特徵展現出卓越的潛力來應對這些任務。然而，這類模型的開發和部署通常需要機器學習和軟體工程方面的專業知識，這為模型的應用帶來了障礙。在本文中，我們提出了 CodeTF，一個開源的基於 Transformer 的庫，用於最先進的程式碼LLMs和程式碼智能。遵循模塊化設計和可擴展框架的原則，我們設計了 CodeTF，具有統一的接口，以實現對不同類型的模型、數據集和任務的快速訪問和開發。我們的庫支持一系列預訓練的程式碼LLM模型和流行的程式碼基準，包括一個標準化接口，以有效地訓練和提供程式碼LLMs，以及語言特定解析器和用於提取程式碼屬性的實用功能。在本文中，我們描述了設計原則、架構、關鍵模塊和組件，並與其他相關庫工具進行了比較。最後，我們希望 CodeTF 能夠彌合機器學習/生成式人工智能與軟體工程之間的鴻溝，為開發人員、研究人員和從業者提供全面的開源解決方案。

English

Code intelligence plays a key role in transforming modern software engineering. Recently, deep learning-based models, especially Transformer-based large language models (LLMs), have demonstrated remarkable potential in tackling these tasks by leveraging massive open-source code data and programming language features. However, the development and deployment of such models often require expertise in both machine learning and software engineering, creating a barrier for the model adoption. In this paper, we present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Following the principles of modular design and extensible framework, we design CodeTF with a unified interface to enable rapid access and development across different types of models, datasets and tasks. Our library supports a collection of pretrained Code LLM models and popular code benchmarks, including a standardized interface to train and serve code LLMs efficiently, and data features such as language-specific parsers and utility functions for extracting code attributes. In this paper, we describe the design principles, the architecture, key modules and components, and compare with other related library tools. Finally, we hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering, providing a comprehensive open-source solution for developers, researchers, and practitioners.