AdamW: Decoupled Weight Decay Regularization

Ilya Loshchilov and Frank Hutter (University of Freiburg), Decoupled Weight Decay Regularization, ICLR 2019; arXiv:1711.05101, v1 14 November 2017, read from v3 (4 January 2019) as the PDF. The paper behind the optimizer most deep-learning frameworks now default to. Until today this spoke held it only as one line in wikipedia-stochastic-gradient-descent. Closes coverage edge 7's successor.

The argument

For plain SGD, L2 regularization and weight decay are the same thing once the decay factor is rescaled by the learning rate. For Adam they are not, and the paper says this is "often overlooked." Adding an L2 term to the loss puts the penalty's gradient through Adam's per-parameter scaling, so "weights with large historic parameter and/or gradient amplitudes" end up regularized less than true weight decay would regularize them. The common libraries of the time implemented L2 and called it weight decay, which the authors call misleading.

Their explanation for a known puzzle follows from that: Adam often generalized worse than SGD with momentum on image classification because, on those tasks, the regularizer that helps SGD was not actually reaching Adam's weights.

The fix

Apply the decay directly to the weights, outside the gradient step, as in the original definition of weight decay. Algorithm 2 sets Adam with L2 and AdamW side by side, and the difference is where the λθ term goes. Two further proposals:

• Normalized weight decay. The best decay factor shrinks as the training budget grows. The paper replaces λ with λ_norm and sets λ = λ_norm · √(b / BT) (batch size b, B training points, T epochs), so one setting carries across budgets.
• AdamWR. Cosine annealing with warm restarts (from the same authors' SGDR) added to AdamW. Adam's per-parameter adaptation, the paper argues, does not remove the gain from a global schedule.

What it measured

26-layer 2x64d and 2x96d ResNets on CIFAR-10 and ImageNet32x32, budgets from 100 to 1800 epochs. At the default learning rate of 0.001, AdamW gave a 15% relative improvement in test error over Adam on both datasets. Decoupling also made the best learning rate and the best decay factor close to independent, so they can be tuned one at a time, for SGD and for Adam. AdamWR reached the same results with up to 10× better anytime performance at the first restart. Together these closed most of the gap to SGD with warm restarts on CIFAR-10 and matched it on ImageNet32x32.

The caveat this spoke records

Same shape as adam-optimizer: the authors of the method ran the comparison, on two image datasets they chose, with one architecture family. As with Adam, the wider check came through adoption rather than a benchmark. The paper lists PyTorch and TensorFlow implementations made by others before publication, and decoupled decay is now the default in most frameworks. The image-classification results do not by themselves show that the fix matters for the language-model training where AdamW is now standard. That use rests on practice, not on this paper.

Related

adam-optimizer · stochastic-gradient-descent · wikipedia-stochastic-gradient-descent · gradient-descent · synthesis

Cross-spoke. The training context sits in ../machine-learning-wiki.