基于检索增强搜索的大语言模型程序优化

摘要

近期研究展示了大型语言模型（LLMs）在程序优化方面的潜力，这是编程语言领域的一项关键挑战。我们提出了一种称为检索增强搜索（RAS）的黑盒自适应方法，该方法对候选优化方案执行束搜索；在每一步中，它从给定的慢速-快速程序对训练数据集中检索上下文示例，以引导LLM。关键的是，我们发现基于LLM生成的自然语言描述进行上下文检索，其效果显著优于基于源代码的检索。此外，我们还提出了AEGIS方法，通过将训练示例分解为本质上更具增量性的“原子编辑”来提高可解释性。实验表明，在优化C++程序时，RAS的性能比先前最先进的黑盒自适应策略高出2.06倍，而AEGIS在实现更小规模编辑的同时，性能提升可达1.37倍。我们还证明，使用RAS可将Python程序的平均运行时间百分位数相比基线提升10.27。

English

Recent work has demonstrated the potential of large language models (LLMs) for program optimization, a key challenge in programming languages. We propose a blackbox adaptation method called Retrieval Augmented Search (RAS) that performs beam search over candidate optimizations; at each step, it retrieves in-context examples from a given training dataset of slow-fast program pairs to guide the LLM. Critically, we find that performing contextual retrieval based on an LLM-generated natural language description significantly outperforms retrieval based on the source code. We also propose AEGIS, a method for improving interpretability by decomposing training examples into ''atomic edits'' that are significantly more incremental in nature. We show that RAS performs up to 2.06times better than prior state-of-the-art blackbox adaptation strategies on optimizing C++ programs, and that AEGIS performs up to 1.37times better while making significantly smaller edits. We also show that using RAS improves the mean runtime percentile of Python programs by 10.27 compared to baselines.