Bayesian optimization
Bayesian optimization (BO) is the wiki’s first model-based optimizer — a sharp contrast to the model-free metaheuristics that make up the rest of the corpus. It targets global optimization of expensive-to-evaluate, black-box functions where evaluations cost real time/money and derivatives are unavailable. Pioneered by Jonas Mockus (1970s–80s) and popularized by Donald R. Jones (1998). Source: Wikipedia (independent of andrey-dik).
How it works
Two parts work in a loop:
- Surrogate model — usually a Gaussian process — a probabilistic fit to all observations so far, giving both a prediction and an uncertainty everywhere.
- Acquisition function — expected improvement, probability of improvement, or upper confidence bound — picks the next point to sample, “trad[ing] off exploration and exploitation so as to minimize the number of function queries.”
So exploration-vs-exploitation here is explicit and principled (sample where improvement is likely or uncertainty is high), not emergent from population dynamics.
Why it matters here — a new axis
BO reframes the whole corpus along a sample-efficiency / model axis:
- Model-free metaheuristics (genetic-algorithm, particle-swarm-optimization, cma-es, …) spend many cheap evaluations exploring blindly.
- BO spends few expensive evaluations, reasoning about where to look from a learned model.
Different regime, different winner — the no-free-lunch-theorem across evaluation budget, not just landscape. Its flagship use is ML hyperparameter tuning (adopted at Google, Meta, OpenAI).
Related
metaheuristic-optimization · exploration-vs-exploitation · no-free-lunch-theorem · cma-es · differential-evolution · test-functions-for-optimization