LLM-Driven Kernel Discovery for High-Dimensional Bayesian Optimization
A new framework called Kernel Discovery uses large language models (LLMs) to automate the design of Gaussian Process kernels for high-dimensional Bayesian optimization. The approach overcomes two key bottlenecks: limited kernel search spaces in existing automated methods (restricted to additions and multiplications of base kernels) and infeasibility of LLM-based methods that condition on raw observations (due to context-length limits and difficulty extracting patterns). The framework employs a two-stage evolutionary process where an LLM first proposes kernel code, then refines it without needing observation conditioning, enabling search over a broader space of kernel compositions.
Key facts
- Kernel Discovery is an LLM-driven evolutionary framework for high-dimensional Bayesian optimization.
- It searches a broader kernel space beyond predefined composition rules.
- It does not require conditioning on observations.
- Existing automated approaches struggle due to limited kernel search spaces.
- LLM-based approaches conditioning on raw observations are infeasible in high dimensions.
- The framework uses a two-stage approach: LLM proposes kernel code, then refines it.
- Directly prompting an LLM yields syntactically varied but functionally identical kernels.
- The work is published on arXiv with ID 2605.20249.
Entities
Institutions
- arXiv