Adam, AdaGrad, RMSprop: Adaptive Learning Rates

Introduction

In momentum-based methods, we use a single learning rate \eta for all parameters. But neural networks have millions of parameters, and they're not created equal:

• Parameters connected to frequent features (common words, bright pixels) get big, stable gradients.
• Parameters connected to rare features (unusual words, edge cases) get small, noisy gradients.

The Core Insight

Adaptive optimizers give each parameter its own learning rate, automatically tuned based on gradient history. This normalizes the optimization landscape and lets rare features catch up.

The Problem with Global Learning Rate

Consider training a word embedding model. The word "the" appears millions of times; its gradient is massive and stable. The word "serendipity" appears twice; its gradient is tiny and unreliable.

If LR is Large

"the" overshoots and oscillates. "serendipity" finally learns something.

If LR is Small

"the" converges nicely. "serendipity" barely moves in a lifetime of training.

The Solution

Divide the learning rate by the "magnitude" of recent gradients. Big gradients get small effective LR. Small gradients get large effective LR. The playing field is leveled.

AdaGrad (Adaptive Gradient, 2011)

AdaGrad (Duchi et al., 2011) was the first breakthrough for sparse data problems. It maintains a sum of squared gradients for each parameter:

Accumulator

G_{t} = G_{t - 1} + g_{t}^{2}

Parameter Update

\theta_{t + 1} = \theta_{t} - \frac{\eta}{\sqrt{G_{t} + \epsilon}} \cdot g_{t}

How It Works

• Frequent features: Large G_{t} so effective LR = \eta/\sqrt{G_{t}} is small.
• Rare features: Small G_{t} so effective LR stays large.
• \epsilon (typically 10^{- 8} prevents division by zero.

AdaGrad's Strength

Excellent for sparse data (NLP, click-through prediction). Rare events get aggressive updates when they finally appear, compensating for their infrequency.

Interactive: Learning Rate Adaptation

See how AdaGrad adapts the learning rate differently for frequent vs rare features. Watch the accumulator grow and the effective learning rate change.

Adaptive Learning Rate

Visualizing AdaGrad's decay based on feature frequency.

Frequent "The"

Rare "Serendipity"

Accumulated Gradient (G)

162.04

Sum of squared gradients. Grows rapidly due to frequent updates.

Effective Learning Rate

0.0393

\eta_{eff} = \eta/\sqrt{G_{t}} Decays quickly to prevent oscillation.

"The word "the" appears constantly. AdaGrad brakes hard to stop it from exploding."

AdaGrad's Fatal Flaw

AdaGrad has a critical problem: the accumulator G_{t} is a sum of positive numbers. It only grows, never shrinks.

The Learning Rate Death Spiral

As training progresses:

\left. G_{t}\rightarrow\infty\;\,\Longrightarrow\;\,\frac{\eta}{\sqrt{G_{t}}}\rightarrow 0 \right.

The effective learning rate decays to zero. The model freezes before reaching the optimum. Training simply stops making progress.

This is acceptable for convex problems (you're near the optimum anyway). But for deep learning's non-convex landscapes, you need to keep exploring. AdaGrad gives up too early.

RMSprop (Root Mean Square Propagation)

RMSprop was proposed by Geoff Hinton in a Coursera lecture (Lecture 6e). It's never been formally published, yet it powers much of modern AI.

The fix is simple: instead of a cumulative sum, use an exponential moving average (EMA) of squared gradients. The accumulator "forgets" ancient history.

Leaky Accumulator

E\lbrack g^{2}\rbrack_{t} = \beta E\lbrack g^{2}\rbrack_{t - 1} + (1 - \beta)g_{t}^{2}

Parameter Update

\theta_{t + 1} = \theta_{t} - \frac{\eta}{\sqrt{E\lbrack g^{2}\rbrack_{t} + \epsilon}} \cdot g_{t}

Why EMA Works

With \beta = 0.9 the accumulator averages roughly the last 10 squared gradients. It doesn't grow to infinity; it stabilizes around the recent average magnitude. Learning continues indefinitely.

Adam (Adaptive Moment Estimation)

Adam (Kingma & Ba, 2015) combines the best of Momentum and RMSprop:

First Moment (Mean)

Like Momentum: smooths the gradient direction.

m_{t} = \beta_{1}m_{t - 1} + (1 - \beta_{1})g_{t}

Second Moment (Variance)

Like RMSprop: scales the learning rate.

v_{t} = \beta_{2}v_{t - 1} + (1 - \beta_{2})g_{t}^{2}

Adam Update

\theta_{t + 1} = \theta_{t} - \frac{\eta}{\sqrt{{\hat{v}}_{t}} + \epsilon} \cdot {\hat{m}}_{t}

where {\hat{m}}_{t} and {\hat{v}}_{t} are bias-corrected moments

Default Hyperparameters

Learning Rate

\eta = 0.001

First Moment Decay

\beta_{1} = 0.9

Second Moment Decay

\beta_{2} = 0.999

Interactive: Optimizer Race

Watch all four optimizers race through a ravine. Notice how AdaGrad slows down over time while RMSprop and Adam maintain speed.

Adaptive Optimizers Race

Comparing trajectories in a high-curvature landscape.

Start Race

Reset

Steps: 0

Live Performance

SGD

Loss:25.74000

AdaGrad

Stops early (LR→0)

Loss:25.74000

RMSprop

Leaky accumulator

Loss:25.74000

Adam

Adaptive + Momentum

Loss:25.74000

Bias Correction Deep-Dive

Adam initializes m_{0} = 0 and v_{0} = 0 This creates a problem in early training.

The Initialization Bias

At step 1, with \beta_{2} = 0.999

v_{1} = 0.999(0) + 0.001 \cdot g_{1}^{2} = 0.001g_{1}^{2}

The estimate is 1000× smaller than the true squared gradient! Without correction, the learning rate would explode.

The Fix

Divide by (1 - \beta^{t})(1−βt) to scale up early estimates:

{\hat{m}}_{t} = \frac{m_{t}}{1 - \beta_{1}^{t}}

{\hat{v}}_{t} = \frac{v_{t}}{1 - \beta_{2}^{t}}

At t=1: 1 - 0.999^{1} = 0.001 Dividing by 0.001 multiplies by 1000, exactly compensating for the bias.

Interactive: Bias Correction

See the bias correction in action. Toggle it on/off to understand why it's necessary in early training.

Bias Correction

Addressing the zero-initialization problem in Moving Averages.

Correction ON

Time Step (t)

60

Correction Factor

1.00×

\frac{1}{1 - \beta^{t}} (where \beta = 0.9

Uncorrected

0.998

Biased towards 0

Corrected

1.000

Matches True (1.0)

"Without correction, Adam starts too slow. The factor boosts early estimates to match reality."

AdamW: Decoupled Weight Decay

For years, researchers noticed that Adam generalized worse than SGD+Momentum on vision tasks. Loshchilov & Hutter (2017) discovered the culprit: incorrect implementation of L2 regularization.

The Problem: L2 Regularization vs Weight Decay

In SGD, L2 regularization and weight decay are mathematically equivalent. Adding \frac{\lambda}{2}\parallel w\parallel^{2} to the loss produces gradient \nabla L + \lambda w which after the update gives:

w_{t + 1} = w_{t} - \eta(\nabla L + \lambda w_{t}) = (1 - \eta\lambda)w_{t} - \eta\nabla L

The term (1 - \eta\lambda)w_{t}(1−ηλ)wt​ is weight decay. For SGD, adding L2 to the loss and applying weight decay directly produce the same result. But for Adam, they diverge.

Standard Adam + L2

Adds \lambda w to the gradient before adaptive scaling:

m_{t} = \beta_{1}m_{t - 1} + (1 - \beta_{1})(g_{t} + \lambda w_{t})

v_{t} = \beta_{2}v_{t - 1} + (1 - \beta_{2})(g_{t} + \lambda w_{t})^{2}

Problem: The weight decay term \lambda w gets scaled by 1/\sqrt{v_{t}} Parameters with large gradients receive less regularization.

AdamW (Decoupled)

Applies weight decay after the Adam update, not through the gradient:

m_{t} = \beta_{1}m_{t - 1} + (1 - \beta_{1})g_{t}

v_{t} = \beta_{2}v_{t - 1} + (1 - \beta_{2})g_{t}^{2}

w_{t + 1} = w_{t} - \eta\frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + \epsilon} - \eta\lambda w_{t}

Why Decoupling Matters

Uniform Regularization

In AdamW, every parameter receives the same relative weight decay \eta\lambda regardless of gradient magnitude. This matches the intended behavior of L2 regularization.

Hyperparameter Independence

The optimal weight decay \lambda becomes independent of the learning rate \eta In standard Adam+L2, you need to retune \lambda whenever you change \eta

Better Generalization

AdamW achieves generalization comparable to SGD+Momentum while retaining Adam's fast convergence. This closed the gap that made practitioners prefer SGD for vision tasks.

AdamW for Transformers

AdamW is the default optimizer for BERT, GPT, ViT, and virtually all modern transformers. Typical settings: \eta = 1e\text{-}4 to 3e\text{-}4 \lambda = 0.01 to 0.1 \beta_{1} = 0.9 \beta_{2} = 0.999 If you're training a transformer, use AdamW.

Adam vs AdamW

X has 20x steeper gradients. Watch how regularization differs.

Steps: 0

Step: 0

Adam + L2

Loss:66.240

Effective \lambda_{x}

Effective \lambda_{y}

\lambda_{x} < \lambda_{y} : steep direction gets less regularization

AdamW

Loss:66.240

\lambda_{x}

\lambda_{y}

\lambda_{x} = \lambda_{y} : uniform decay, better generalization

The Problem

Adam+L2 divides weight decay by \sqrt{v} Steep directions (large v get weaker regularization. AdamW keeps decay uniform.

The Adam Controversy

Adam is not universally loved. There's an ongoing debate about when to use it.

Adam Wins: Fast Convergence

Adam converges faster in early training. Great for prototyping, NLP, and when compute is limited. It's forgiving of learning rate choice.

SGD+Momentum Wins: Better Generalization

Many vision papers report that SGD+Momentum finds flatter minima that generalize better to test data. The noise in SGD acts as implicit regularization.

The Practical Compromise

Many practitioners use Adam for early training (reach a good region fast), then switch to SGD for fine-tuning (find a flat minimum). Some use learning rate warmup to help Adam's early instability.

Comparison Table