基於檢索增強搜尋的大型語言模型程式最佳化

摘要

近期研究顯示大型語言模型（LLMs）在程式最佳化（程式語言領域的一項關鍵挑戰）中具有潛力。我們提出名為「檢索增強搜尋」（RAS）的黑箱適應方法，該方法對候選最佳化方案執行波束搜尋；每一步皆從給定的慢速－快速程式配對訓練資料集中檢索上下文範例，以引導LLM。關鍵的是，我們發現基於LLM生成的自然語言描述進行上下文檢索，其表現顯著優於基於原始碼的檢索。此外，我們提出AEGIS方法，透過將訓練範例分解為本質上更具漸進性的「原子編輯」，提升可解釋性。我們證明RAS在最佳化C++程式方面，效能比先前最先進的黑箱適應策略高出2.06倍；而AEGIS在進行顯著較小幅度編輯的同時，效能提升達1.37倍。同時，使用RAS能將Python程式的平均運行時間百分位數較基準方法提升10.27。

English

Recent work has demonstrated the potential of large language models (LLMs) for program optimization, a key challenge in programming languages. We propose a blackbox adaptation method called Retrieval Augmented Search (RAS) that performs beam search over candidate optimizations; at each step, it retrieves in-context examples from a given training dataset of slow-fast program pairs to guide the LLM. Critically, we find that performing contextual retrieval based on an LLM-generated natural language description significantly outperforms retrieval based on the source code. We also propose AEGIS, a method for improving interpretability by decomposing training examples into ''atomic edits'' that are significantly more incremental in nature. We show that RAS performs up to 2.06times better than prior state-of-the-art blackbox adaptation strategies on optimizing C++ programs, and that AEGIS performs up to 1.37times better while making significantly smaller edits. We also show that using RAS improves the mean runtime percentile of Python programs by 10.27 compared to baselines.