用於編譯器優化的大型語言模型

摘要

我們探索了將大型語言模型應用於程式碼優化的新方法。我們提出了一個由頭開始訓練的 70 億參數的 Transformer 模型，用於優化 LLVM 組合語言以縮小程式碼大小。該模型將未經優化的組合語言作為輸入，並輸出一組最佳優化該程式的編譯器選項。在訓練過程中，我們要求模型預測優化前後的指令計數，以及優化後的程式碼。這些輔助學習任務顯著提高了模型的優化性能，並增強了模型對問題的理解深度。我們在一組龐大的測試程式上進行評估。我們的方法在減少指令計數方面比編譯器提高了 3.0%，優於需要數千次編譯的兩種最先進的基準方法。此外，該模型展現出驚人的程式碼推理能力，能夠在 91% 的情況下生成可編譯的程式碼，並在 70% 的情況下完美模擬編譯器的輸出。

English

We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after optimization, and the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and improve the model's depth of understanding. We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler, outperforming two state-of-the-art baselines that require thousands of compilations. Furthermore, the model shows surprisingly strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time.

用於編譯器優化的大型語言模型

Large Language Models for Compiler Optimization

摘要

Support