Bayesian optimization
Bayesian optimization (BO) is the wiki’s first model-based optimizer — a sharp contrast to the model-free metaheuristics that make up the rest of the corpus. It targets global optimization of expensive-to-evaluate, black-box functions where evaluations cost real time/money and derivatives are unavailable. Pioneered by Jonas Mockus (1970s–80s) and popularized by Donald R. Jones (1998). Source: Wikipedia (independent of andrey-dik).
How it works
Two parts work in a loop:
- Surrogate model — usually a Gaussian process — a probabilistic fit to all observations so far, giving both a prediction and an uncertainty everywhere.
- Acquisition function — expected improvement, probability of improvement, or upper confidence bound — picks the next point to sample, “trad[ing] off exploration and exploitation so as to minimize the number of function queries.”
So exploration-vs-exploitation here is explicit and principled (sample where improvement is likely or uncertainty is high), not emergent from population dynamics.
Why it matters here — a new axis
BO reframes the whole corpus along a sample-efficiency / model axis:
- Model-free metaheuristics (genetic-algorithm, particle-swarm-optimization, cma-es, …) spend many cheap evaluations exploring blindly.
- BO spends few expensive evaluations, reasoning about where to look from a learned model.
Different regime, different winner — the no-free-lunch-theorem across evaluation budget, not just landscape. Its flagship use is ML hyperparameter tuning (adopted at Google, Meta, OpenAI). Textbook treatments file it under surrogate models rather than under metaheuristics algorithmsbook — the branch is named for what it builds, not for how it searches.
Related
metaheuristic-optimization · exploration-vs-exploitation · no-free-lunch-theorem · cma-es · differential-evolution · test-functions-for-optimization · algorithmsbook