optimization-algorithms-wiki

CMA-ES (Covariance Matrix Adaptation Evolution Strategy)

CMA-ES is the theory-heavyweight of the founding four: a population-based evolution strategy developed by Nikolaus Hansen & Andreas Ostermeier (late 1990s), widely regarded as a top general-purpose continuous optimizer. Source: andrey-dik‘s MQL5 implementation (C_AO_CMAES).

How it works

Samples candidate solutions from a multivariate normal x_k ~ N(m, σ²C) (mean m, step-size σ, covariance C), then each iteration:

• Ranks candidates by fitness, selects the best μ;
• updates the mean by weighted recombination of top performers;
• adapts the covariance matrix via rank-one and rank-μ updates along evolutionary paths;
• adapts step-size by monitoring path length (stagnation detection).

Strengths & limits

• Invariance to affine transforms of the search space: “equally efficiently solves f(x) and f(Ax + b)” — excellent on ill-conditioned, noisy, discontinuous, multi-modal landscapes.
• Scalability wall: O(n²) memory, O(n³) operations — “for high-dimensional problems (n > 100), the resource intensity becomes disproportionate.”

Benchmark (context-relative)

Ranked ~38/45 (48.33%) on the population-optimization-benchmark — fine at 5D/25D, failed at 500D on cost. A striking case for the no-free-lunch-theorem: high reputation, mid-low rank on this suite (see synthesis).

Caveat

Benchmark figure is from one MQL5 suite/implementation; CMA-ES rates far higher on many academic benchmarks — read the rank as suite-relative.

Related

metaheuristic-optimization · exploration-vs-exploitation · no-free-lunch-theorem · population-optimization-benchmark · andrey-dik · backtracking-search-algorithm