Nelder–Mead method
The derivative-free direct-search classic the corpus’s quadrant map was missing — proposed by John Nelder & Roger Mead (1965), a.k.a. the downhill simplex (“amoeba”) method. It finds a local optimum by comparing function values only (no gradients, no model), occupying a quadrant distinct from both the metaheuristics and the gradient/exact methods.
Mechanism
It maintains a simplex of n+1 vertices in n-dimensional space (a triangle in 2D, a tetrahedron in 3D) and iteratively replaces its worst vertex via four geometric moves: reflection (flip the worst point through the opposite face), expansion (push further when reflection helps a lot), contraction (pull inward when stuck), and shrink (collapse toward the best point). The simplex crawls/rolls downhill as one body.
Where it sits in the field
On the synthesis’s axes it stakes out a specific corner:
- Single-solution, non-population — like simulated-annealing and tabu-search, it evolves one state (a simplex), not a population; but unlike them it’s a direct-search geometry, not a sampling metaheuristic.
- Derivative-free but local & largely deterministic — it shares “no gradients” with the metaheuristics, but it is local (no global-search ambition) and not stochastic, contrasting with the random exploration of genetic-algorithm/ particle-swarm-optimization and with the derivative-using gradient-descent.
- Cheap, few evaluations — practical for expensive, low-dimensional objectives, which is also its niche vs. model-based bayesian-optimization (BO models the surface; Nelder–Mead just probes it).
The No-Free-Lunch read
Its documented weaknesses are textbook no-free-lunch-theorem: it “can converge to non-stationary points” on problems other methods solve cleanly, degrades badly in high dimensions, and is sensitive to the initial simplex — strong on cheap smooth low-dim problems, poor where structure or scale defeats pure geometry. Another illustration that each optimizer trades assumptions for reach (convex-optimization‘s structure⇄generality).
Related
metaheuristic-optimization · simulated-annealing · tabu-search · gradient-descent · bayesian-optimization · no-free-lunch-theorem