Wikipedia — Stochastic gradient descent

Authoritative reference used to deepen stochastic-gradient-descent with the stochastic-approximation provenance, convergence conditions, and the named optimizer lineage.

What it adds

• Provenance. “In 1951, Herbert Robbins and Sutton Monro introduced the earliest stochastic approximation methods, preceding stochastic gradient descent.” SGD is a special case of Robbins–Monro stochastic approximation.
• Convergence. “when the learning rates η decrease with an appropriate rate … stochastic gradient descent converges almost surely to a global minimum” for convex/pseudoconvex objectives.
• Mini-batches. “A compromise … is to compute the gradient against more than one training sample (called a ‘mini-batch’)” — the basis of efficient vectorized training.
• Adam provenance. “Adam (short for Adaptive Moment Estimation) is a 2014 update to the RMSProp optimizer combining it with the main feature of the Momentum method”, by Kingma & Ba (2014); AdaGrad (2011) and RMSProp (2012) precede it.
• AdamW decouples weight decay from the gradient-based update, improving regularization.

Tier T2, freshness volatile (the “dominant optimizer” claim drifts). Cited from stochastic-gradient-descent.