Wikipedia — Gradient descent

Authoritative reference used to deepen gradient-descent with provenance, convergence rates, and the conditioning story.

What it adds

• Provenance. “Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847.” Hadamard proposed a similar method in 1907; convergence was first studied by Haskell Curry (1944).
• Convergence rates. On convex, Lipschitz-smooth functions plain gradient descent achieves O(1/k) error; Nesterov acceleration improves this to O(1/k²), which is optimal for first-order methods.
• Conditioning. Iteration count is “commonly proportional to the spectral condition number κ(A)”; ill-conditioning produces “a characteristic zig–zag path” with slow convergence.
• Step sizes. The Barzilai–Borwein rule sets the step from successive iterate/gradient differences; step sizes meeting the Wolfe conditions (via line search) guarantee convergence.
• Momentum. The “heavy ball method” adds a momentum term “in analogy to a heavy ball sliding on the surface”, matching conjugate-gradient rates on quadratics.

Tier T2. Cited from gradient-descent.