Adam: A Method for Stochastic Optimization

Kingma & Ba, ICLR 2015 (arXiv:1412.6980). The primary source for the optimizer that almost every neural network on earth is trained with. Until now this spoke held it only at second hand, through wikipedia-stochastic-gradient-descent summarising it in two sentences. Closes coverage edge 7.

The update rule

Adam keeps two exponential moving averages per parameter: m of the gradient (first moment) and v of the squared gradient (second raw moment, the uncentered variance). Both start at zero, which biases them toward zero early on, so both are divided by 1 − β^t before use. That bias correction is the paper's smallest and most consequential piece of arithmetic. The step is then θ ← θ − α · m̂ / (√v̂ + ε).

Defaults the paper gives for the problems it tested: α = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸. Those four numbers have survived essentially unchanged into every major framework since.

Why it belongs in a metaheuristics wiki

The step size is what the paper actually argues about, and the argument is one this corpus has not seen made before. The effective step Δ = α · m̂/√v̂ is bounded by α in the ordinary case, and invariant to the scale of the gradient — rescale every gradient by c and the c cancels. Kingma and Ba read that as "establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information," which makes α a quantity you can pick from knowing the geometry of the problem rather than by tuning. The ratio m̂/√v̂ they call a signal-to-noise ratio: as it approaches an optimum the SNR falls and the steps shrink on their own — automatic annealing, arrived at by a different route than simulated-annealing's explicit cooling schedule.

The relationship to its neighbours is stated precisely rather than gestured at. AdaGrad is Adam with β₁ = 0, infinitesimal (1 − β₂) and an α·t^(−1/2) annealing — a special case, not a cousin. RMSProp with momentum differs in two ways: it applies momentum to the rescaled gradient, and it has no bias correction, which the paper says matters most when β₂ is near 1 (the setting sparse gradients need) because the uncorrected estimate then produces very large steps and often diverges.

What it measured

Logistic regression on MNIST and on IMDB bag-of-words (10,000 sparse features, 50% dropout), multilayer neural networks, and deep convolutional networks, with hyper-parameters searched over a dense grid for every optimizer compared and the best setting reported. On MNIST logistic regression Adam converges much like SGD with Nesterov momentum and both beat AdaGrad; on the sparse IMDB features the ordering flips, which is the point of running both. Section 4 gives a regret bound "comparable to the best known results under the online convex optimization framework": a guarantee in the convex regime only, which is not the regime anyone runs Adam in.

The caveat this spoke is obliged to record

Every comparison in the paper was run by the authors of the method being compared, scored on benchmarks they chose. That is the corpus-wide pattern flagged in quality-log.md — the same structure as DOAm leading Dik's suite, and as nsga-ii beating PAES and SPEA on metrics its own authors defined. Adam differs in one respect worth stating: the replication happened anyway, at a scale no benchmark provides, because the whole field adopted it and kept it. The gap the no-free-lunch-theorem frame cares about is narrower than usual here, and it is still a gap.

See adamw for the weight-decay correction that frameworks now default to, stochastic-gradient-descent for the lineage and gradient-descent for the deterministic parent.