用于编译器优化的大型语言模型

摘要

我们探讨了大型语言模型在代码优化中的新应用。我们展示了一个从头开始训练的包含70亿参数的Transformer模型，用于优化LLVM汇编以减小代码大小。该模型接受未经优化的汇编作为输入，并输出一系列最佳优化程序的编译器选项。在训练过程中，我们要求模型预测优化前后的指令计数以及优化后的代码本身。这些辅助学习任务显著提高了模型的优化性能，并增强了模型的理解深度。我们在大量测试程序上进行评估。我们的方法在减少指令计数方面比编译器提高了3.0%，胜过了需要数千次编译的两种最先进基线。此外，该模型表现出惊人的强大代码推理能力，91%的时间生成可编译代码，并在70%的时间内完美模拟编译器的输出。

English

We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after optimization, and the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and improve the model's depth of understanding. We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler, outperforming two state-of-the-art baselines that require thousands of compilations. Furthermore, the model shows surprisingly strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time.

用于编译器优化的大型语言模型

Large Language Models for Compiler Optimization

摘要

Support