回歸式程式碼語言模型

摘要

我們研究代碼到指標的回歸：預測代碼執行的數值結果，這是一項因編程語言的開放性而具有挑戰性的任務。雖然先前的方法依賴於繁重且特定領域的特徵工程，但我們展示了一個單一的統一回歸語言模型（RLM）能夠同時從文本中直接預測：(i) 跨多種高級語言（如Python和C++）的代碼內存佔用，(ii) Triton GPU內核的延遲，以及(iii) 以ONNX表示的訓練神經網絡的準確性和速度。特別是，一個相對較小的300M參數的RLM，從T5Gemma初始化，在APPS的競賽編程提交中獲得了>0.9的斯皮爾曼等級相關性，並且在CodeNet的17種不同語言中，單一統一模型的平均斯皮爾曼等級相關性>0.5。此外，RLM在五個經典的NAS設計空間中獲得了最高的平均肯德爾-陶係數0.46，這些空間先前由圖神經網絡主導，並且能夠同時預測多種硬件平台上的架構延遲。

English

We study code-to-metric regression: predicting numeric outcomes of code executions, a challenging task due to the open-ended nature of programming languages. While prior methods have resorted to heavy and domain-specific feature engineering, we show that a single unified Regression Language Model (RLM) can simultaneously predict directly from text, (i) the memory footprint of code across multiple high-level languages such as Python and C++, (ii) the latency of Triton GPU kernels, and (iii) the accuracy and speed of trained neural networks represented in ONNX. In particular, a relatively small 300M parameter RLM initialized from T5Gemma, obtains > 0.9 Spearman-rank on competitive programming submissions from APPS, and a single unified model achieves > 0.5 average Spearman-rank across 17 separate languages from CodeNet. Furthermore, the RLM can obtain the highest average Kendall-Tau of 0.46 on five classic NAS design spaces previously dominated by graph neural networks, and simultaneously predict architecture latencies on numerous hardware platforms.

回歸式程式碼語言模型

Regression Language Models for Code

摘要

Support