Session S06

Optimizers

Use a gradient to choose the next parameter values.

SGD · momentum · RMSProp · Adam · AdamW · schedules

Deep Learning · CS 4715/5715 · University of Idaho

Backward supplies a gradient array; the optimizer uses it

\[W\in\mathbb R^{D\times H},\qquad G=\nabla_W L\in\mathbb R^{D\times H}\]

\[W^{\mathrm{new}}=W-\eta G,\qquad\eta>0\ \text{is the learning rate}\]

For each optimizer: identify the arrays, write out the update rule, and translate it into code. Derivations appear right after the concept they derive.

Read the operation before translating it into code

In optimizer updates, nearly all operations are elementwise: the parameter, gradient, and state arrays share the same shape. Squares and square roots also act elementwise.

SGD (stochastic gradient descent) applies one array update

Stochastic means the gradient is computed from a random minibatch, not all training data. In practice, “SGD” refers to minibatch SGD.

\[W=\begin{bmatrix}2&2\\1&-1\end{bmatrix},\quad G=\begin{bmatrix}2&50\\1&-25\end{bmatrix},\quad\eta=0.03\]

The gradient G is given. Scale it first, then subtract entry by entry:

\[\eta G=\begin{bmatrix}0.06&1.50\\0.03&-0.75\end{bmatrix},\qquad W^{\mathrm{new}}=W-\eta G=\begin{bmatrix}1.94&0.50\\0.97&-0.25\end{bmatrix}\]

W = np.array([[2.0, 2.0], [1.0, -1.0]])
G = np.array([[2.0, 50.0], [1.0, -25.0]])
W_new = W - 0.03 * G

Minibatch sampling makes the gradient array noisy

• Different minibatches can give different gradients at the same parameter values.
• With persistent gradient noise, a fixed learning rate can leave late-training fluctuations.
• Learning-rate decay reduces step size later. Momentum remembers past gradients.

From here on, the bowl: \(L(w)=\tfrac12\left(w_0^2+25\,w_1^2\right)\), curvatures 1 and 25, start \((2,2)\), full gradients. Curvature is isolated from minibatch noise.

One rate, many curvatures: the bound is per coordinate

\[L(w)=\tfrac12\textstyle\sum_i c_iw_i^2\quad\Rightarrow\quad w_i^{\mathrm{new}}=(1-\eta c_i)\,w_i\]

No cross term, so each coordinate contracts on its own. S03 derived \(0<\eta<2/c_i\) for one coordinate; a shared \(\eta\) must stay below every such bound, so the largest curvature binds.

\[0<\eta<\min_i\frac{2}{c_i}=\frac{2}{c_{\max}}=\frac{2}{25}=0.08\]

The largest curvature caps \(\eta\); at that cap the smallest curvature converges slowly: 0.25 per step on the steep coordinate, 0.97 on the shallow one. Plain GD on this quadratic only.

The ratio that sets the speed: \(\kappa=c_{\max}/c_{\min}\)

Stability caps the rate at \(2/c_{\max}\); progress along the shallow coordinate goes as \(1-\eta c_{\min}\). Balance them and both coordinates contract equally:

\[\eta^\star=\frac{2}{c_{\min}+c_{\max}}\quad\Rightarrow\quad|1-\eta^\star c_{\min}|=|1-\eta^\star c_{\max}|=\frac{\kappa-1}{\kappa+1}\]

On this bowl \(\eta^\star=2/26\approx0.077\), giving \(24/26\approx0.923\): the best a single rate can do.

Iterations scale with \(\kappa\): the slowness is the shape of the bowl, and no value of \(\eta\) changes it.

What \(\kappa\) is, and what it is not

• \(\kappa\) is the ratio of the largest to the smallest curvature. This bowl is diagonal, so each coordinate is a curvature direction and \(\kappa\) reads off the two values. A cross term rotates the slow direction off the axes: same \(\kappa\), aligned with no coordinate.
• In a network the curvatures change with \(w\) and need not all be positive, so \(\kappa\) is a local guide, not a constant of the problem.

On this bowl the slow and fast directions are the two axes, so one step size per axis works. A cross term rotates the slow direction off the axes, and per-coordinate scaling no longer targets it. Read “this optimizer fixes conditioning” as true only for axis-aligned conditioning.

Momentum keeps a velocity array with the parameter shape

\[W,G,V\in\mathbb R^{D\times H},\qquad V_0=0\]

\[V_t=\beta V_{t-1}+G_t,\qquad\beta=0.9,\qquad W^{\mathrm{new}}=W-\eta V_t\]

\(\beta\) is the momentum coefficient: the fraction of the previous velocity that is kept.

The current gradient can change sign while stored velocity still reflects earlier gradients. The next page traces an overshoot.

Momentum code: update the state, then the parameter

def momentum(w, g, s, t):
    v = 0.9 * s["v"] + g
    return w - lr * v, {"v": v}

For a constant gradient, this unnormalized velocity approaches \(g/(1-0.9)=10g\). Changing gradients need not follow that ratio.

Momentum accelerates the slow coordinate

Trace the shallow coordinate: start \(w=2,\ v_0=0\), with \(g=1\cdot w\), \(\eta=0.03\).

Here \(g=w\), so each step’s gradient is the previous w. After 3 steps SGD reaches 1.825 and momentum 1.672: the velocity accumulates gradients that keep the same sign, so the effective step grows.

Momentum accelerates directions where gradients point the same way across steps. The steep coordinate can overshoot for the same reason: the next page traces that case.

Momentum: the current gradient joins a stored velocity

\[v_t=0.9v_{t-1}+g_t,\qquad w^{\mathrm{new}}=w-0.03v_t\]

Deriving: whether momentum overshoots on the steep coordinate. Start \(w=2,\ v_0=0\), with \(g=25w\).

Result: at step 3 the gradient is negative, but the stored velocity still points in the previous direction. Momentum overshoots here, but the old velocity decays by 0.9 each step, so the overshoot is temporary. This is the cost of the acceleration shown on the preceding page.

AdaGrad (adaptive gradient): scale each entry by its history

\[R_t=R_{t-1}+G_t\odot G_t,\qquad R_0=0\]

\[W^{\mathrm{new}}=W-\eta\,G_t\oslash(\sqrt{R_t}+\epsilon)\]

\(R\) is the running sum of squared gradients, entry by entry; \(\epsilon\) is a small positive constant that keeps the denominator nonzero. Same bowl, \(\eta=0.03\):

\(R\) only grows, so the step shrinks toward zero and never recovers, even if the gradient changes. The two columns agree because dividing by \(\sqrt{R}\) removes the gradient’s scale.

RMSProp remembers gradient scale for every parameter entry

\[R_t=\beta_2R_{t-1}+(1-\beta_2)(G_t\odot G_t),\qquad\beta_2=0.99,\qquad R_0=0\]

\[W^{\mathrm{new}}=W-\eta\,G_t\oslash(\sqrt{R_t}+\epsilon)\]

Same update as AdaGrad; only how \(R\) accumulates changes. This is an exponential moving average: each step multiplies the old squares by \(\beta_2\), so they fade instead of piling up, and the step can grow again after a large gradient has passed.

The displayed RMSProp has no momentum buffer. Its state describes gradient size; the numerator carries the current sign.

Expand the moving average to see the weights on past values

Unroll \(R_t = \beta R_{t-1} + (1{-}\beta)\,x_t\):

\[\begin{aligned}R_t &= (1{-}\beta)\,x_t + \beta\,R_{t-1}\\&= (1{-}\beta)\,x_t + \beta(1{-}\beta)\,x_{t-1} + \beta^2(1{-}\beta)\,x_{t-2} + \cdots\end{aligned}\]

The weight on \(x_{t-k}\) is \(\beta^k(1{-}\beta)\). These weights sum to 1 and decay exponentially.

RMSProp uses \(\beta_2=0.99\): the squared-gradient average looks back about 100 steps. Adam uses \(\beta_1=0.9\) for gradients (10 steps) and \(\beta_2=0.999\) for squared gradients (1000 steps).

RMSProp code: each operation matches one calculation

def rmsprop(w, g, s, t):
    r = 0.99 * s["r"] + 0.01 * g * g
    return w - lr * g / (np.sqrt(r) + 1e-8), {"r": r}

• g * g squares each gradient entry.
• np.sqrt(r) converts the stored squared scale back to gradient units.
• 1e-8 keeps the denominator positive; it does not repair NaN inputs.
• Return the updated r so the next call remembers it.

RMSProp: work through its first step

Deriving: RMSProp’s first step on the bowl and why it exceeds \(\eta\). Inputs \(w_0=(2,2)\), \(g=(2,50)\), \(r_0=0\), \(\beta_2=0.99\), \(\eta=0.03\).

After 3 steps SGD reaches \([1.83,\,0.03]\) and RMSProp \([1.36,\,1.36]\): SGD moved the shallow coordinate only 0.17 while the steep one is at 0.03; RMSProp moved both equally.

Result: the first step is \(0.3=10\eta\) on both coordinates, because the zero-initialized average gives \(\sqrt{R_1}=0.1|g|\).

Adam (adaptive moment estimation) maintains two state arrays

\[M_t=0.9M_{t-1}+0.1G_t,\qquad M_0=0\]

\[V_t=0.999V_{t-1}+0.001(G_t\odot G_t),\qquad V_0=0\]

The coefficients are Adam’s defaults \(\beta_1=0.9\) and \(\beta_2=0.999\); note that \(\beta_2\) differs from RMSProp’s 0.99.

Code uses m and v for these arrays. Adam’s v is a squared-gradient average; momentum’s v was a velocity.

Adam: correct the state arrays, then compute the update

\[\widehat M_t=\frac{M_t}{1-0.9^t},\qquad\widehat V_t=\frac{V_t}{1-0.999^t}\]

\[W^{\mathrm{new}}=W-\eta\,\widehat M_t\oslash(\sqrt{\widehat V_t}+\epsilon)\]

• The scalar correction factors broadcast across every state entry.
• Why correct: \(M_0=0\) gives \(M_1=0.1G_1\), 10% of the gradient; dividing by \(1-0.9^1=0.1\) restores the scale.
• Start \(t\) at 1. Save \(M_t,V_t\); use their corrected versions for this update.

\(M\) and \(V\) start biased toward zero by different amounts (\(\beta_1=0.9\) vs \(\beta_2=0.999\)), so the ratio \(M/\!\sqrt{V}\) is distorted: page 22 shows an uncorrected first step of \(3.16\eta\). Correction makes each moment unbiased, so \(\eta\) controls the step size as intended.

Adam code: moments, correction, update

def adam(w, g, s, t):
    m = 0.9 * s["m"] + 0.1 * g
    v = 0.999 * s["v"] + 0.001 * g * g
    m_hat = m / (1 - 0.9 ** t)
    v_hat = v / (1 - 0.999 ** t)
    return w - lr * m_hat / (np.sqrt(v_hat) + 1e-8), {"m": m, "v": v}

• Start t at 1; t = 0 would divide by zero.
• The hats are used to compute this step. Save the uncorrected m and v for the next recurrence.
• The caller supplies lr, NumPy, current weights, gradients, and state.

The caller keeps the parameter matrix and state together

curv = np.array([1.0, 25.0])
def grad(W):
    return W * curv  # broadcast over the columns

W = np.array([[2.0, 2.0]])  # one-row parameter matrix
state = {"m": np.zeros_like(W), "v": np.zeros_like(W)}
lr = 0.03  # read by adam() as a module-level variable
for t in range(1, 201):
    G = grad(W)
    W, state = adam(W, G, state, t)

This two-coordinate bowl supplies an analytic gradient array. A network supplies G through backward. The adam function reads lr from the enclosing scope; torch.optim.Adam receives it as a constructor argument instead.

Initialize once, recompute G at the current W, and retain both returned values. Evaluate loss after the update.

Adam bias correction: calculate step 1

Deriving: the size of Adam’s first step with and without bias correction, using the shallow coordinate’s first gradient \(g_1=2\) and both memories initially zero.

\[\frac{\hat m_1}{\sqrt{\hat v_1}}=\frac{2}{2}=1,\qquad |\Delta w|\approx\eta=0.03\]

Result: corrected, the first step is about \(\eta\); uncorrected, \(0.2/\sqrt{0.004}\approx3.16\), about \(3.16\eta\). That ratio is \((1-\beta_1)/\sqrt{1-\beta_2}\) for any \(g_1\): a property of the defaults, not of this example. Tiny \(\epsilon\) omitted.

Compare progress on a two-coordinate quadratic

\[L(W)=\tfrac12\operatorname{sum}(W\odot W\odot c),\quad c=[1,25],\quad W_0=[2,2]\]

Elementwise products, then sum all entries · full gradients · \(\eta=0.03\) for every method · momentum \(\beta=0.9\) · RMSProp \(\beta_2=0.99\) · Adam \((0.9,\,0.999)\) · \(\epsilon=10^{-8}\) · initial loss 52.

Which method leads at update 10? Does that establish a universal winner?

The ranking depends on the checkpoint and shared rate. Printed zeros are rounded. Compare method + hyperparameters + budget.

Replace the hand-written function with torch.optim

opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
for epoch in range(1, 401):
    opt.zero_grad()                     # clear stored gradients
    loss = loss_fn(model(X), y)         # forward
    loss.backward()                     # backward
    opt.step()                          # update all parameters

Ring classification: 2→32→32→2 ReLU, cross-entropy, full batch, 400 updates, shared init.

Same loop for every optimizer; only the constructor changes. One run, not a generalization benchmark.

L2 regularization and weight decay

Add a penalty to the loss that grows with the size of the weights:

\[L_{\text{total}}=L_{\text{data}}+\frac\lambda2\lVert W\rVert_F^2\]

• The penalty \(\frac\lambda2\lVert W\rVert_F^2\) sums the squares of every weight entry. Larger weights increase the total loss.
• Its gradient is \(\lambda W\): each weight is pushed toward zero in proportion to its current value.
• In the update, this produces a factor \((1-\eta\lambda)\) that shrinks \(W\) every step. This shrinking is called weight decay.

S07 measures what this penalty does to validation error. Here we need only the gradient \(\lambda W\) to see how Adam and AdamW handle it differently.

Use the L2 gradient as an array rule

The penalty gradient is \(\lambda W\), the same shape as the parameter matrix:

\[R(W)=\frac\lambda2\lVert W\rVert_F^2,\qquad\nabla_W R=\lambda W\]

For plain SGD without momentum, the updates coincide. In Adam, coupled L2 changes both moment arrays. The SGD equivalence is derived on the next page.

Substitute the matrix gradient into plain SGD

Deriving: the plain-SGD update with the L2 penalty, to locate the factor \((1-\eta\lambda)\).

\[\widetilde G=G+\lambda W\]

\[W^{\mathrm{new}}=W-\eta(G+\lambda W)\]

\[W^{\mathrm{new}}=(1-\eta\lambda)W-\eta G\]

• Distribute the scalar learning rate, then collect the two terms containing W.
• This equality uses plain SGD without momentum. Coupled L2 in Adam first changes its moment arrays.

Result: under plain SGD, coupled L2 equals shrinking \(W\) by \(1-\eta\lambda\) and then taking an ordinary gradient step. That shrink is weight decay.

AdamW applies decay directly to the parameter matrix

\[W^{\mathrm{new}}=(1-\eta\lambda)W-\eta\,\widehat M\oslash(\sqrt{\widehat V}+\epsilon)\]

• The scalar factor \(1-\eta\lambda\) multiplies every entry of the old \(W\).
• The moment arrays use the data gradient; decay stays outside them.
• Use torch.optim.AdamW for this decoupled update.

Check decay against a zero-decay control

Two weights start at 1.0. Their data gradients alternate ±30 and ±0.3. Use lr = 0.01 for 500 updates. With \(\lambda=0\) Adam and AdamW are identical, so the control row applies to both.

\[\text{Pure decay only: }(1-0.01\times0.1)^{500}\approx0.6064\]

Coupled L2 inside Adam decays the two weights unequally, 0.9662 against 0.1859: after normalization, \(\lambda W\) is large relative to a small data gradient. AdamW shrinks both by the same factor, 0.5958. The control shows why that is not 0.6064: alternating raw gradients still produce net adaptive motion.

A schedule changes the rate used by each update

Base rate 3×10⁻⁴ · 100 warmup updates · 1000 total updates.

This schedule starts at rate 0, reaches the base rate at index 100, then follows one cosine decay.

Read the schedule as a multiplier of the base rate

total_steps, warmup_steps = 1000, 100
def lr_lambda(step):
    if step < warmup_steps:
        return step / warmup_steps
    progress = min(1.0, (step - warmup_steps) / 900)
    return 0.5 * (1.0 + math.cos(math.pi * progress))
sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)

Keep gradient computation, updates, and scheduling in order

opt.zero_grad()
loss = loss_fn(model(x_batch), y_batch)
loss.backward()
opt.step()       # use the current learning rate
sched.step()     # prepare the rate for the next update

• Here the scheduler advances once per optimizer update.
• Clear gradients once per update group; accumulation across minibatches requires deliberate scaling.
• With this zero-start schedule, the first parameter step uses rate 0. Adam’s state can still update.

Check the mechanism before copying a recipe

Can you explain why resetting optimizer state every update changes the algorithm?

The saved gradient history disappears. The next step no longer implements the intended recurrence.

Check the derivations before moving on

On the bowl \(c=[1,25]\) with \(\eta=0.03\), what multiplies the steep coordinate under plain GD?

\(1-\eta c=1-0.03\times25=0.25\); positive, so it contracts without overshooting.

Why can momentum still overshoot after the gradient changes sign?

The stored velocity still points in the old direction and can outweigh the reversed gradient.

Which Adam values are stored between updates?

The uncorrected \(M\) and \(V\); the hats are recomputed each step.

For next time

• Read: S07: Regularization, Initialization, and Normalization.
• S07: control fitting, choose initial signal scale, and identify normalization groups.
• HW1 is due at S08: scalar autograd operations, backward ordering, and a gradient-descent training loop.
• Quiz 1 is at S09: all of Module 1, including today’s optimizers.