编译器对大型语言模型的优先采样

摘要

大型语言模型在生成和优化代码方面表现出巨大潜力。广泛使用的采样方法，如核采样，增加了生成的多样性，但在低温度下往往会产生重复的样本，在高温度下会产生不连贯的样本。此外，温度系数必须针对每个任务进行调整，限制了其可用性。我们提出了优先采样，这是一种简单且确定性的采样技术，可按照模型的置信度产生有序的独特样本。每个新样本都会扩展增广搜索树中具有最高概率的未扩展标记。此外，优先采样支持基于正则表达式的生成，提供可控且结构化的探索过程。优先采样在任意数量的样本上优于核采样，将原始模型的性能从2.87%提升至5%以上。此外，仅需30个样本，优先采样就优于用于生成原始模型训练标签的自动调谐器。

English

Large language models show great potential in generating and optimizing code. Widely used sampling methods such as Nucleus Sampling increase the diversity of generation but often produce repeated samples for low temperatures and incoherent samples for high temperatures. Furthermore, the temperature coefficient has to be tuned for each task, limiting its usability. We present Priority Sampling, a simple and deterministic sampling technique that produces unique samples ordered by the model's confidence. Each new sample expands the unexpanded token with the highest probability in the augmented search tree. Additionally, Priority Sampling supports generation based on regular expression that provides a controllable and structured exploration process. Priority Sampling outperforms Nucleus Sampling for any number of samples, boosting the performance of the original model from 2.87% to 5% improvement over -Oz. Moreover, it outperforms the autotuner used for the generation of labels for the training of the original model in just 30 samples.

编译器对大型语言模型的优先采样

Priority Sampling of Large Language Models for Compilers

摘要

Support