利用大语言模型实现贝叶斯优化的自适应核设计如同小菜一碟

摘要

贝叶斯优化（BO）的效率在很大程度上依赖于高斯过程（GP）核的选择，该核在有限的评估预算下对探索与利用的平衡起着核心作用。传统的BO方法通常依赖于固定或启发式的核选择策略，当所选核与目标函数不匹配时，可能导致收敛缓慢或次优解。为解决这一局限，我们提出了一种新颖的上下文感知核进化方法（CAKE），通过大型语言模型（LLMs）增强BO。具体而言，CAKE利用LLMs作为交叉和变异算子，在整个优化过程中根据观测数据自适应地生成并优化GP核。为最大化CAKE的效能，我们进一步提出了BIC-获取核排序（BAKER），通过平衡由贝叶斯信息准则（BIC）衡量的模型拟合度与每次BO迭代的期望改进，选择最有效的核。大量实验表明，基于CAKE的新BO方法在一系列实际任务中，包括超参数优化、控制器调谐和光子芯片设计，均显著优于现有基线。我们的代码已公开于https://github.com/cake4bo/cake。

English

The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/cake4bo/cake.

利用大语言模型实现贝叶斯优化的自适应核设计如同小菜一碟

Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

摘要

Support