教授大型語言模型低資源語言：增強Pharo中的程式碼補全

摘要

大型語言模型（LLMs）為自動化程式碼撰寫開啟了新的可能性，並成為多數程式碼補全工具的基礎。雖然 LLMs 在主流程式語言上表現出色，但它們往往缺乏對所謂低資源語言（訓練資料稀少）的支援，導致這些語言的程式碼補全工具品質落後於它們所服務的社群。具體實例為 Pharo，這是一種受 Smalltalk 啟發的語言，其 IDE 目前僅提供單一語彙單元（token）的補全功能。在本研究中，我們報告了將基於 LLM 的程式碼補全引入 Pharo 的經驗。首先，我們描述了一個端到端管線，該管線結合了 Pharo 專屬的資料策展、繼續預訓練以及開源程式碼 LLM 的微調。其次，我們介紹了一系列 Pharo 程式碼補全基準測試，旨在評估模型是否（i）學會 Pharo 的語法，以及（ii）能準確補全來自真實 GitHub 儲存庫的遮蔽 Pharo 程式碼。第三，我們以實證數據顯示，Pharo 特化模型不僅顯著優於其原始基礎檢查點，且在 Pharo 補全任務上準確度也超過了規模更大的程式碼 LLM。整體而言，我們的個案研究證明了將強效的基於 LLM 的程式碼補全帶入低資源程式語言的可行性，且所使用的模型足夠小巧，能在 IDE 內提供「即時」支援。

English

Large Language Models (LLMs) unlocked new possibilities in automated code writing, becoming the backbone of most code completion tools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality of code completion tooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-based code completion to Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation, continued pre-training and fine-tuning of open code LLMs. Second, we introduce a set of Pharo code completion benchmarks designed to evaluate whether models (i) learn Pharo's syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-based code completion to low-resource programming languages, with models small enough to provide ``real-time'' in-IDE support.