Session S03

Loss Functions and Gradient Descent

Cross-entropy · Numerical stability · Learning rates

Deep Learning — CS 4715/5715 · University of Idaho

One recipe to train them all

• The perceptron rule (S02) was special-purpose: one model, one kind of mistake
• Modern replacement: write a loss \(L(\theta)\) — one number measuring wrongness — and descend it:

\[\theta \;\leftarrow\; \theta - \eta\,\nabla_\theta L(\theta)\]

• The same machinery trains a 2-parameter regression and a 100B-parameter language model
• \(L\) must be differentiable (almost everywhere) — accuracy is a staircase in the parameters, flat almost everywhere: nothing to descend

Today's models: one scorer, three heads

The scorer is S02's linear model — \(z = \mathbf{w}^\top\mathbf{x} + b\), a score computed from features. An output head turns the score into a prediction, and the head picks the loss:

• This family is what the reading's code actually runs: fit the straight line \(y = 2x + 1\); scan a sigmoid head's score; calibrate an nn.Linear
• Deep networks (S04 →) swap the scorer and keep the heads and losses — nothing today gets thrown away.

Gradient descent — the name of the recipe

• The gradient \(\nabla_\theta L\) is a vector with one slope per parameter: \(\big(\partial L/\partial\theta_1,\; \partial L/\partial\theta_2,\; \dots\big)\)
• It points in the direction of steepest local increase of the loss — uphill
• So step against it, scaled by the learning rate \(\eta\): the steepest local decrease — gradient descent

\[L\big(\theta - \eta\,\nabla_\theta L\big) \;\approx\; L(\theta) - \eta\,\lVert\nabla_\theta L\rVert^2 \;\le\; L(\theta)\] (first-order Taylor, small \(\eta\))

• A squared norm is never negative — for small enough \(\eta\), every step lowers the loss
• "Small enough" is load-bearing: the \(\approx\) fails when \(\eta\) is too large — the learning-rate laboratory measures exactly where.

Where losses come from: maximum likelihood

• Assume a probabilistic model of how targets arise given inputs; choose the parameters that maximize the probability of the training labels:

\[\theta^\star \;=\; \arg\max_\theta \prod_i p_\theta(y_i \mid x_i) \;=\; \arg\min_\theta \Big[-\sum_i \log p_\theta(y_i \mid x_i)\Big]\]

• Every loss today is \(-\log p\) under a chosen probability model of the labels — the assumption is the loss
• Gaussian noise around a predicted value → MSE (next slide); Bernoulli labels → binary cross-entropy; categorical labels → cross-entropy
• This bridge is the line the whole session hangs off — each loss we meet today reads back through it.
• The flip side: every loss designed this way carries a strong assumption about the error distribution — most papers use the loss and forget the assumption.

Regression: mean squared error

\[\text{MSE} \;=\; \frac{1}{N}\sum_i (\hat{y}_i - y_i)^2\]

One forward pass, identity head, on the straight line the reading's code fits — \(y = 2x + 1\), so \(w = 2,\, b = 1\): \[x = 3 \;\xrightarrow{\;wx + b\;}\; \hat{y} = 7, \qquad \text{label } y = 5, \qquad L = (7-5)^2 = 4\]

• Identity head: the score is the prediction; the forward pass computes the prediction, then the loss
• The square is not arbitrary: assume \(y \mid x \sim \mathcal{N}(\hat{y}, \sigma^2)\), take \(-\log p\), and MSE falls out — minimizing MSE = maximum likelihood under Gaussian noise
• When that assumption is wrong (corrupted labels, outliers) MSE misleads — MAE (absolute value instead of the square) and Huber trade the property away; the reading's table compares all three
• Nothing has been learned yet — the loss is just a measurement. Improving it is the next page.

One gradient step: update the parameters, not the prediction

The prediction is not a knob we can turn — the parameters are. Chain rule, then one step with \(\eta = 0.025\): \[\frac{\partial L}{\partial w} = \underbrace{2(\hat{y} - y)}_{=\,4}\cdot\, x = 12, \qquad \frac{\partial L}{\partial b} = 2(\hat{y} - y)\cdot 1 = 4\] \[w \leftarrow 2 - 0.025 \times 12 = 1.7, \qquad b \leftarrow 1 - 0.025 \times 4 = 0.9\] \[\text{forward again:}\quad \hat{y} = 1.7 \times 3 + 0.9 = 6, \qquad L: \; 4 \to (6-5)^2 = 1\]

• Both parameter gradients reuse the same upstream factor \(2(\hat{y} - y) = 4\): the error drives the update — big mistakes, big corrections
• One step, and the loss measurably dropped. That is the entire recipe, working, on the smallest possible example
• For classification the modern default is cross-entropy over softmax — best motivated by watching the alternative fail

Why not MSE for classification? (reading 1.3)

Sigmoid head now: \(\mathbf{x} \xrightarrow{\;\mathbf{w}^\top\mathbf{x} + b\;} z \xrightarrow{\;\sigma\;} p \xrightarrow{\;\text{loss}\;} L\) — the table scans the scorer's output \(z\) for one example with true label \(y = 1\):

# One example with true label y = 1, prediction p = sigmoid(z).
# MSE loss:            L = (p - y)^2   -> dL/dz = 2 * (p - y) * p * (1 - p)
# Binary cross-entropy: L = -log(p)    -> dL/dz = p - y
      

z (logit)   p=sigmoid(z)   |dL/dz| MSE   |dL/dz| CE
  -8.0         0.0003      0.000670      0.9997
  -4.0         0.0180      0.034690      0.9820
  -2.0         0.1192      0.184956      0.8808
   0.0         0.5000      0.250000      0.5000
   2.0         0.8808      0.025031      0.1192
      

First row: the model is as wrong as it gets (\(p = 0.0003\) on the true class) — and MSE's gradient is ~1000× smaller than binary cross-entropy's. MSE punishes confident errors with a whisper.

Matched pairs: the cancellation

• MSE's culprit: the chain-rule factor \(p(1-p)\) — sigmoid's own derivative, vanishing in both saturated regimes
• Binary cross-entropy through sigmoid: the sigmoid derivative cancels exactly against the log
• Result \(\partial L/\partial z = p - y\): push proportional to the probability error, largest when the model is most wrong
• Sigmoid + binary cross-entropy (and softmax + cross-entropy) are matched pairs; the cancellation is the content of the match
• You have seen the destination; the next two slides walk the road.

The cancellation — the two pieces

Binary cross-entropy on a sigmoid output: \(p = \sigma(z)\), \(y \in \{0, 1\}\), \(L = -\big[y \log p + (1-y)\log(1-p)\big]\)

Two independent facts, nothing combined yet. Step 0 is the exact factor that killed MSE; step 1 is the log's contribution — each term differentiates to a reciprocal.

The cancellation — multiply, and watch

• \(\partial L/\partial z = p - y\): push proportional to the probability error, largest when most wrong
• MSE keeps the factor the log devoured: \(\partial L/\partial z = 2(p - y)\,p\,(1-p)\) — the whisper on confident errors

Matched pairs — why this is the point

• sigmoid + binary cross-entropy and softmax + cross-entropy are matched pairs: the cancellation you just watched is the content of the match
• Minimizing cross-entropy is maximizing log-likelihood — the maximum-likelihood page's promise, cashed
• An output sigmoid should saturate on confident predictions; saturation is not the bug — the matched loss keeps the gradient healthy anyway
• The pairing comes from maximum likelihood, not from convention.

Softmax: from scores to probabilities

\[\mathbf{p} = \operatorname{softmax}(\mathbf{z}), \qquad p_k \;=\; \frac{e^{z_k}}{\sum_j e^{z_j}}\]

• Vector in, vector out: exponentiate every score (positive, amplifies differences), then normalize (sums to 1)
• Shift invariance: \(\operatorname{softmax}(\mathbf{z} + c) = \operatorname{softmax}(\mathbf{z})\) — only differences matter (we lean on this hard, twice)
• Loss for true class \(y\): \(\;L = -\log p_y\); gradient w.r.t. the logits: \(\;\nabla_{\mathbf{z}} L = \mathbf{p} - \operatorname{onehot}(y)\)
• Name and pairing entered the literature ~1990 via John Bridle. Slightly off name: it is a soft argmax. The machinery is older — the inverse link of multinomial logistic regression.

Cross-entropy, properly: one formula

And the multiclass gradient, for the softmax head? Next page, line by line.

The multiclass cancellation — expand the loss

Softmax head now — multiclass, say ten animal classes. The goal: the same clean gradient the sigmoid case earned.

• Softmax is a fraction with exponentials — so the loss starts as a nested log-of-a-quotient
• Log of a quotient: \(\log(A/B) = \log A - \log B\); the log meets \(e^{z_y}\) and eats the exp, leaving \(z_y\)
• One nested expression became two easy parts. Now differentiate them — next page.

The multiclass cancellation — differentiate, term by term

• \(\mathbf{1}[k=y]\) is 1 when \(k\) is the true class, else 0 — the one-hot entry (\(-z_y\) only reacts when \(k = y\))
• The second term, by the chain rule, is \(e^{z_k}\) over the sum — exactly the softmax definition, \(p_k\)
• Prediction minus truth — the same \(p - y\) as the sigmoid case, one cancellation per class
• And \(\log \sum_j e^{z_j}\) is about to solve a numerical-stability problem — remember it.

Softmax: a worked number

Softmax head: three scorers, one per class — \(z_k = \mathbf{w}_k^\top\mathbf{x} + b_k\) — produced this input's scores:

• Which logit gets pushed up? The true class — the only negative gradient entry
• The same \(p - y\) shape as the binary case — one cancellation per class

The overflow you will hit (reading 1.3)

def softmax_naive(z):
    e = np.exp(z)
    return e / e.sum()

def softmax_stable(z):
    e = np.exp(z - z.max())         # shift so the largest logit is 0
    return e / e.sum()

z = np.array([15.0, 1000.0, -8.0])  # logits like this appear in real runs
      

naive softmax:  [ 0. nan  0.]
stable softmax: [0. 1. 0.]
naive  cross-entropy for class 0: inf
stable cross-entropy for class 0: 985.0000
      

exp overflows float64 just past \(z = 709\); naive: \(e^{1000} = \infty\), then \(\infty/\infty = \texttt{nan}\) — and one nan poisons everything downstream. Subtracting the max is legal because only differences matter.

Log-sum-exp, act two: the underflow

• Even the stable softmax printed [0. 1. 0.] — where did class 0's probability go?
• The true value is \(e^{-985}\) — far below the smallest positive float64 (about \(10^{-308}\)), so it rounds to exactly \(0\)
• And then the loss is \(-\log 0 = \infty\) — stability fix number one bought us a second failure
• But the unrepresentable probability has a perfectly ordinary log: the log of \(e^{-985}\) is just \(-985\)
• So never form the probability — compute its log directly (next slide)

Log-sum-exp: the identity, step by step

• The shift license again — only differences between logits matter — now applied inside the log
• After the shift, the largest exponent in the sum is \(e^{0} = 1\): nothing can overflow
• Does it survive the poison logits? The check is the next page.

Log-sum-exp: the check, on the poison logits

• \(985.0000 = 1000 - 15\), the logit gap — clean, finite, exact
• Every serious framework fuses the ops: PyTorch's nn.CrossEntropyLoss takes raw logits; softmax-then-log by hand reintroduces the underflow

Gradient descent: the learning-rate laboratory (reading 1.3)

# 2-D quadratic bowl, deliberately ill-conditioned:
# L(w) = 0.5 * (w[0]**2 + 25 * w[1]**2); grad = [w[0], 25 * w[1]]
for lr in [0.005, 0.06, 0.081]:
    w = np.array([10.0, 1.0])
    for step in range(101):
        w = w - lr * grad(w)
      

lr = 0.005   loss at steps 0/25/50/75/100: 62.5, 38.93, 30.29, 23.57, 18.35
lr = 0.06    loss at steps 0/25/50/75/100: 62.5, 2.267, 0.1027, 0.004657, 0.0002111
lr = 0.081   loss at steps 0/25/50/75/100: 62.5, 43.7, 147.7, 507.6, 1745
      

The idea goes back to Cauchy (1847); everything practical hides in the learning rate \(\eta\) (lr in code).

Three regimes, three signatures

• Crawling (lr = 0.005): loss creeps — training "works" but wastes your compute budget
• Converging (lr = 0.06): loss collapses by orders of magnitude
• Exploding (lr = 0.081): each step overshoots the valley floor, lands higher on the opposite wall — geometric divergence
• For this bowl the boundary is exact: convergence if and only if \(\eta < 2/25 = 0.08\) (25 = curvature of the steepest direction)
• You will read these three signatures off your own loss curves all semester. The next three slides earn the boundary with one line of algebra.

The divergence bound — one update, factored

• One update multiplies \(w\) by the constant \(1 - \eta a\) — linear in \(w\); the constant does not depend on \(w\), so every later step multiplies by the same thing. Next page

The divergence bound — t steps are geometric

• The same constant multiplies \(w\) at every step — that observation is the entire derivation
• Careful: \(r\) can be negative (\(\eta a > 1\)) — then \(w\) flips sign every step. That is where oscillation comes from
• One ratio \(r\) decides everything — crawling, converging, exploding: three sizes of \(|r|\). The map is the next page

The divergence bound — the boundary, and the bowl

\[|1 - \eta a| < 1 \;\iff\; 0 < \eta a < 2 \;\iff\; \boxed{\;\eta < \frac{2}{a}\;}\]

• \(0 < \eta a < 1\): monotone decay · \(1 < \eta a < 2\): sign flips, still shrinks · \(\eta a > 2\): sign flips and grows — each step lands higher on the opposite wall
• The bowl has no cross term, so it decouples: each coordinate scales by its own ratio (\(1 - \eta\) and \(1 - 25\eta\)) — which is why the 2-D path curves even though each coordinate only scales. One \(\eta\) serves both, so the steepest direction binds
• \(\eta < 2/25 = 0.08\) — why 0.06 converged and 0.081 exploded
• Exact only for quadratics — real losses give you the three signatures, not a formula.

Two practical amendments

• Minibatches: the exact gradient averages the whole training set — wasteful when a random batch of 32–512 gives an unbiased, if noisy, estimate at a fraction of the cost
• That is SGD (roots: Robbins & Monro, 1951) — "gradient descent" in deep learning almost always means minibatch SGD
• Schedules: the right \(\eta\) changes as training proceeds — typically decayed over time (→ S06)
• Léon Bottou's long-running case: with abundant data, fast noisy steps beat slow exact ones — the currency is accuracy per second of compute, not per pass over the data (reading 1.3: batch 32 vs a 32,000-example set is ~30× noisier, 1000× cheaper).

Debugging table: symptoms you will see

For next time

• Read: 1.4 Backpropagation and Computation Graphs
• S04: Backpropagation I — computation graphs and the chain rule
• HW1 (micro-autograd) goes out next session (S04), due S08 — the first graded work of the semester
• The \(\operatorname{softmax} - \operatorname{onehot}\) gradient derived today is the first step of every backward pass on Wednesday.