S03 Reading — Losses and Gradient Descent | Deep Learning

Module 1 · Reading 1.3 · read before S03: Loss Functions and Gradient Descent

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

The perceptron of section 1.2 learned by a special-purpose rule that only makes sense for one model and one kind of mistake. Modern deep learning replaces that with a single, universal recipe: write down a loss function (one number measuring how wrong the model currently is) and adjust every parameter a little in the direction that decreases it. Once training is “minimize a differentiable loss,” the same machinery trains a two-parameter regression and a hundred-billion-parameter language model. The whole rest of this course runs on this recipe.

But the recipe has two places where things silently go wrong, and both are easy to hit in a first implementation. Choose a loss that fights your output nonlinearity and gradients vanish exactly when the model is most wrong. Implement cross-entropy the “obvious” way and it overflows to nan on confident predictions. This section covers the standard losses, why cross-entropy pairs with softmax, the log-sum-exp trick that makes it computable, and gradient descent itself — including how to recognize a learning rate that is too small, right, or catastrophically too large.

How to read this

Losses: one number to descend
Regression losses compared: MSE, MAE, Huber
Why not MSE for classification?
Softmax: from scores to probabilities
The overflow you will hit, and log-sum-exp
Gradient descent and the learning rate
Beyond the bowl: non-convexity
Going deeper
Common pitfalls
Try it yourself
Further reading

Losses: one number to descend

A loss function \(L(\theta)\) maps model parameters to a scalar: average wrongness over the training set. Training is then

\[\theta \;\leftarrow\; \theta - \eta\, \nabla_\theta L\]

repeated until the loss stops improving, where \(\eta\) is the learning rate (lr in every code example). For this to work, \(L\) must be differentiable (almost everywhere) in the parameters, which is the deep reason neural networks use smooth losses over probabilities rather than counting errors: accuracy is a staircase function of the parameters, flat almost everywhere, and gives gradient descent nothing to descend.

This creates a permanent, slightly awkward split you should keep explicit in every project: the metric is what you care about (accuracy, F1, error rate) and the loss is what you can differentiate. The loss is chosen to be a well-behaved surrogate for the metric, and the two usually (not always) improve together. When they diverge (loss falls while accuracy stalls, or vice versa), the divergence itself is diagnostic information, and several of this section’s pitfalls are exactly such divergences.

Where do the standard losses come from? Almost all of them from one principle: maximum likelihood. Assume a probabilistic model of how targets are generated given inputs, then choose parameters that maximize the probability of the training labels; equivalently, minimize the negative log-likelihood. Different noise assumptions yield different losses. Gaussian noise around a predicted value yields mean squared error; a Bernoulli or categorical label distribution yields cross-entropy. The assumption is the loss, which means a loss choice is never neutral: it encodes a claim about how your data misbehaves, and when the claim is wrong the loss misleads — as the next experiment shows concretely.

For regression (predicting real values) the default is mean squared error:

\[\mathrm{MSE} = \frac{1}{N}\sum_i (\hat{y}_i - y_i)^2\]

Its gradient with respect to the prediction, \(2(\hat{y} - y)/N\), grows with the error: big mistakes get big corrections. That sounds like a feature, and under well-behaved Gaussian noise it is. But it also means a single wildly wrong label pulls on the model with force proportional to how wrong it is: MSE is an outlier amplifier.

Regression losses compared: MSE, MAE, Huber

Three regression losses span the practical choices. Writing \(r = \hat{y} - y\) for the residual:

Loss	Formula (per example)	Gradient \(dL/dr\)	Character
MSE	\(r^2\)	\(2r\)	Fastest fit under clean Gaussian-like noise; outliers dominate the gradient
MAE	\(\lvert r\rvert\)	\(\operatorname{sign}(r)\)	Every example pushes with force 1, however wrong; robust but slow near the minimum
Huber	\(\tfrac12 r^2\) if \(\lvert r\rvert \le \delta\), else \(\delta(\lvert r\rvert - \tfrac12\delta)\)	\(\operatorname{clip}(r, -\delta, +\delta)\)	Quadratic for small errors, linear for large: robust and smooth

The gradient column is the one to read. MSE lets a residual of 15 push thirty times harder than a residual of 0.5; MAE gives both the same vote; Huber (proposed by the statistician Peter Huber in a 1964 paper on robust estimation, long before neural networks) caps each example’s pull at \(\delta\). Here is what that means when 8% of your labels are garbage — a data-entry error, a sensor glitch, a mislabeled batch:

import numpy as np

np.random.seed(0)

# True line y = 2x + 1, Gaussian noise -- then 8 of 100 labels corrupted upward
n = 100
x = np.random.uniform(-2, 2, n)
y = 2 * x + 1 + 0.3 * np.random.randn(n)
y[:8] += 15.0

def fit(loss_grad, lr=0.05, steps=4000):
    """Gradient descent on w, b for a loss with per-example residual gradient loss_grad."""
    w, b = 0.0, 0.0
    for _ in range(steps):
        r = w * x + b - y                 # residuals, (100,)
        g = loss_grad(r)                  # dL/dr per example
        w -= lr * np.mean(g * x)
        b -= lr * np.mean(g)
    return w, b

delta = 1.0
losses = {
    "MSE    (grad 2r)":              lambda r: 2 * r,
    "MAE    (grad sign(r))":         lambda r: np.sign(r),
    "Huber  (grad clip(r, +-1))":    lambda r: np.clip(r, -delta, delta),
}
print(f"true line: w = 2.00, b = 1.00   (8 of {n} labels shifted by +15)")
for name, g in losses.items():
    w, b = fit(g)
    print(f"{name:<28} fitted w = {w:.4f}, b = {b:.4f}")

Running this prints:

true line: w = 2.00, b = 1.00   (8 of 100 labels shifted by +15)
MSE    (grad 2r)             fitted w = 2.4592, b = 2.3077
MAE    (grad sign(r))        fitted w = 1.9964, b = 1.1010
Huber  (grad clip(r, +-1))   fitted w = 2.0296, b = 1.1443

Eight bad labels drag the MSE fit’s intercept from 1.0 to 2.31 — the corrupted 8% moved the answer by more than a full unit, because each outlier’s gradient is proportional to its (huge) residual. MAE and Huber, whose per-example gradients are capped, land within 0.15 of the truth. Note what this does not say: MSE is not a bad loss. On clean data it converges faster and is the maximum-likelihood answer. The lesson is the one from the previous subsection: a loss encodes a noise assumption, MSE’s assumption (“errors are small and symmetric”) is violated by outliers, and the loss faithfully executes its wrong assumption. Inspect your residuals; choose accordingly.

For classification, the modern default is cross-entropy over a softmax, and the story of why is best told through the failure of the alternative.

Why not MSE for classification?

Suppose a binary classifier outputs \(p = \operatorname{sigmoid}(z)\), with true label \(y = 1\), and consider training on the logit \(z\), the pre-sigmoid score. Two candidate losses: MSE, \(L = (p - y)^2\), and binary cross-entropy, \(L = -\log p\). Both are minimized at \(p = 1\). They differ in the gradient they send back when the model is confidently wrong (say \(z = -8\), so \(p\) is nearly 0):

import numpy as np

np.random.seed(0)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# One example with true label y = 1, prediction p = sigmoid(z).
# MSE loss:            L = (p - y)^2        -> dL/dz = 2 * (p - y) * p * (1 - p)
# Cross-entropy loss:  L = -log(p)          -> dL/dz = p - y
print("z (logit)   p=sigmoid(z)   |dL/dz| MSE   |dL/dz| CE")
for z in [-8.0, -4.0, -2.0, 0.0, 2.0]:
    p = sigmoid(z)
    g_mse = abs(2 * (p - 1) * p * (1 - p))
    g_ce = abs(p - 1)
    print(f"{z:>6.1f}     {p:10.4f}     {g_mse:9.6f}    {g_ce:8.4f}")

Running this prints:

z (logit)   p=sigmoid(z)   |dL/dz| MSE   |dL/dz| CE
  -8.0         0.0003      0.000670      0.9997
  -4.0         0.0180      0.034690      0.9820
  -2.0         0.1192      0.184956      0.8808
   0.0         0.5000      0.250000      0.5000
   2.0         0.8808      0.025031      0.1192

Read the first row. The model assigns probability 0.0003 to the true class (as wrong as it gets) and MSE’s gradient is 0.00067, a thousand times smaller than cross-entropy’s 0.9997. The culprit is the factor \(p(1-p)\) in MSE’s chain rule: sigmoid’s own derivative, which vanishes in both saturated regimes. MSE punishes confident errors with a whisper. Cross-entropy’s gradient is \(p - y\): the sigmoid derivative cancels exactly against the log, so the push is proportional to the probability error itself, largest precisely when the model is most wrong. This clean \(p - y\) form is not a coincidence; sigmoid + cross-entropy (and softmax + cross-entropy below) are matched pairs, and the cancellation is the mathematical content of the match.

Cross-entropy also has the likelihood pedigree: for a classifier that outputs probabilities, minimizing cross-entropy is maximizing the log-likelihood of the training labels.

One more classification loss deserves a mention because you will meet it in the literature: the hinge loss, \(L = \max(0,\, 1 - ys)\) for labels in \(\{-1, +1\}\) and score \(s\). It is the loss of support vector machines, and its defining property is the opposite of cross-entropy’s: once an example is correctly classified with margin at least 1, its gradient is exactly zero and it stops influencing training entirely. Cross-entropy never fully lets go — a correctly classified example still contributes a small gradient pushing its probability toward 1, which is part of why deep classifiers grow ever more confident the longer you train them (a fact that returns in Going deeper). Deep learning standardized on cross-entropy, mostly because probabilities compose well with the rest of the pipeline; the hinge survives in the margin-based analyses of learning theory.

Softmax: from scores to probabilities

With \(C\) classes, the network produces a vector of logits \(z\) (one score per class), and softmax converts them to probabilities:

\[\operatorname{softmax}(z)_k = \frac{e^{z_k}}{\sum_j e^{z_j}}\]

Exponentiation makes everything positive and amplifies differences; normalization makes the entries sum to 1. Softmax is invariant to adding a constant to all logits (only differences matter), a fact we are about to lean on hard. The cross-entropy loss for true class \(y\) is then \(-\log \operatorname{softmax}(z)_y\), and its gradient with respect to the logits is the same beautiful form as before: \(\operatorname{softmax}(z) - \operatorname{onehot}(y)\).

The name “cross-entropy” deserves one unpacking, because you have now met it in binary and multiclass clothing and they are the same garment. Between a target distribution \(\mathbf{y}\) and a predicted distribution \(\mathbf{p}\), the cross-entropy is \(H(\mathbf{y}, \mathbf{p}) = -\sum_k y_k \log p_k\). With a one-hot target every term but the true class multiplies by zero, and the sum collapses to \(-\log p_y\), exactly the loss above. The binary formula \(-\bigl(y \log p + (1-y)\log(1-p)\bigr)\) is the two-class instance: \(y\) and \(1-y\) are the two slots of a one-hot vector. Keeping the general form in mind pays off later — label smoothing in Going deeper softens the target so every slot contributes. And the cancellation that took a chain rule in the sigmoid case takes two lines here: \(-\log \operatorname{softmax}(z)_y = -z_y + \log\sum_j e^{z_j}\), and differentiating with respect to \(z_k\) gives \(\operatorname{softmax}(z)_k - \operatorname{onehot}(y)_k\): the log eats the exp, one cancellation per class. That \(\log\sum_j e^{z_j}\) term is about to return wearing a numerical-stability hat.

Softmax has one knob worth meeting now because it returns twice more in this course: temperature. Dividing all logits by a scalar \(T\) before the softmax, \(\operatorname{softmax}(z/T)\), leaves the ranking of classes untouched but reshapes the distribution: \(T \lt 1\) sharpens it toward a one-hot spike on the argmax, \(T \gt 1\) flattens it toward uniform, and \(T = 1\) is plain softmax. The name comes from statistical physics, where the same formula (the Boltzmann distribution) describes how energy states are occupied at a given temperature. You will use temperature to repair overconfident probabilities in this section’s Going deeper, and to control randomness when sampling from language models in section 3.7 — same dial, two jobs.

Historical note. Both the name “softmax” and the standard pairing with log-likelihood training entered the neural network literature around 1990 through work of John Bridle, who argued that classifier networks should output probabilities and be trained as statistical models rather than by squared error on 0/1 targets. The name is slightly off: the function is really a soft argmax (it returns a distribution peaked at the largest logit), not a soft maximum. The statistical machinery itself is much older — softmax is the inverse link of multinomial logistic regression, which predates neural networks by decades.

The overflow you will hit, and log-sum-exp

Now the implementation trap. exp overflows float64 just past \(z = 709\). Logits have no reason to stay small — they are unnormalized scores, and as training makes a model confident, they grow. The “obvious” softmax implementation then returns nan, and one nan anywhere poisons every downstream computation, including all your gradients. Here is the failure and the fix side by side:

import numpy as np

np.random.seed(0)

def softmax_naive(z):
    e = np.exp(z)
    return e / e.sum()

def softmax_stable(z):
    e = np.exp(z - z.max())         # shift so the largest logit is 0
    return e / e.sum()

def log_softmax_stable(z):
    m = z.max()
    return z - (m + np.log(np.exp(z - m).sum()))   # log-sum-exp trick

# Logits of this size appear routinely in real training runs
z = np.array([15.0, 1000.0, -8.0])

with np.errstate(over="ignore", invalid="ignore"):   # silence the warnings; inspect the values
    naive = softmax_naive(z)
print(f"naive softmax:  {naive}")
print(f"stable softmax: {softmax_stable(z)}")

# The same failure hits the loss. True class = 0 (logit 15.0):
with np.errstate(over="ignore", invalid="ignore", divide="ignore"):
    loss_naive = -np.log(softmax_naive(z)[0])
loss_stable = -log_softmax_stable(z)[0]
print(f"naive  cross-entropy for class 0: {loss_naive}")
print(f"stable cross-entropy for class 0: {loss_stable:.4f}")

Running this prints:

naive softmax:  [ 0. nan  0.]
stable softmax: [0. 1. 0.]
naive  cross-entropy for class 0: inf
stable cross-entropy for class 0: 985.0000

The naive version computes exp(1000) = inf, then inf / inf = nan. The stable version subtracts the max logit first (legal because softmax only cares about differences), so the largest exponent is exp(0) = 1 and nothing overflows.

The loss lines show a subtler point. Even the stable softmax prints [0. 1. 0.]: the true probability of class 0 is \(e^{-985}\), which underflows to exactly 0 in float64, and -log(0) = inf. Probabilities this small are unrepresentable — but their logs are perfectly ordinary numbers. log_softmax_stable computes the log-probability directly:

\[\operatorname{log\text{-}softmax}(z)_k = z_k - m - \log\sum_j e^{z_j - m}, \qquad m = \max_j z_j\]

and returns a clean \(-985\) for class 0, hence loss 985.0000: exactly \(1000 - 15\), the logit gap. This is the log-sum-exp trick, and it is why every serious framework fuses the two operations: PyTorch’s nn.CrossEntropyLoss takes raw logits and applies log-softmax internally, and calling softmax followed by log yourself reintroduces the underflow the fused version exists to prevent. The exercises below have you implement exactly this stable fused loss and match PyTorch’s output.

Pitfall. The most common numerical-stability bug in practice is not writing a naive softmax — it is mixing conventions: passing already-softmaxed probabilities into a loss that expects logits (double softmax; model trains, badly, and plateaus at mediocre accuracy), or passing logits into a function expecting probabilities (nan within a few steps). When training misbehaves, print one batch’s outputs and check: logits are any real numbers; probabilities are in [0, 1] and sum to 1 per row. Know which one every function in your pipeline expects.

Gradient descent and the learning rate

With a differentiable loss in hand, gradient descent is the update \(\theta \leftarrow \theta - \eta\, \nabla L\). The gradient \(\nabla L\) collects one slope per parameter, \((\partial L/\partial\theta_1,\, \partial L/\partial\theta_2,\, \ldots)\), and points in the direction of steepest local increase of the loss; stepping against it is the steepest local decrease. One line of first-order Taylor says why every small-enough step helps: \(L(\theta - \eta g) \approx L(\theta) - \eta \lVert g\rVert^2\), and a squared norm is never negative. The “small enough” is load-bearing — the approximation fails when \(\eta\) is too large, and the experiment below measures exactly where. The idea is old (it goes back to Cauchy in 1847), but everything practical about deep learning hides in the choice of \(\eta\).

A clean laboratory for learning-rate behavior is a quadratic bowl that is deliberately ill-conditioned — much steeper in one direction than another, which is the normal situation in real networks:

import numpy as np

np.random.seed(0)

# A 2-D quadratic bowl, deliberately ill-conditioned: steep in w[1], shallow in w[0]
# L(w) = 0.5 * (w[0]^2 + 25 * w[1]^2); grad = [w[0], 25 * w[1]]
def loss(w):
    return 0.5 * (w[0] ** 2 + 25 * w[1] ** 2)

def grad(w):
    return np.array([w[0], 25 * w[1]])

for lr in [0.005, 0.06, 0.081]:
    w = np.array([10.0, 1.0])
    trace = []
    for step in range(101):
        if step % 25 == 0:
            trace.append(f"{loss(w):.4g}")
        w = w - lr * grad(w)
    print(f"lr = {lr:<6}  loss at steps 0/25/50/75/100: {', '.join(trace)}")

Running this prints:

lr = 0.005   loss at steps 0/25/50/75/100: 62.5, 38.93, 30.29, 23.57, 18.35
lr = 0.06    loss at steps 0/25/50/75/100: 62.5, 2.267, 0.1027, 0.004657, 0.0002111
lr = 0.081   loss at steps 0/25/50/75/100: 62.5, 43.7, 147.7, 507.6, 1745

Three regimes, three signatures. At lr = 0.005 the loss creeps down: after 100 steps it has not even halved twice; training “works” but wastes your compute budget. At lr = 0.06 the loss collapses by orders of magnitude. At lr = 0.081 the loss rises — each step overshoots the valley floor in the steep direction and lands higher on the opposite wall, diverging geometrically. For this bowl the boundary is exact: steps converge only when \(\eta \lt 2/25 = 0.08\), where 25 is the curvature of the steepest direction. Real losses are not quadratic, but near a minimum they behave like one, and the same three signatures (crawling, converging, exploding) are what you will read off your own loss curves throughout this course. Note also what ill-conditioning does: the workable \(\eta\) is dictated by the steepest direction, which forces slow progress along the shallow one. Momentum and Adam (section 1.6) exist mostly to soften exactly this.

Real training runs add one wrinkle the clean bowl lacks: curvature that varies along the trajectory. A learning rate that was safe for the first thousand steps can hit a sharply curved region and produce one enormous update — visible in your logs as a sudden loss spike, sometimes recoverable, sometimes the first domino before nan. The standard insurance is gradient clipping: before the update, if the gradient’s overall norm exceeds a threshold, rescale it down to that threshold, capping the worst-case step size while leaving typical steps untouched. Global-norm clipping preserves the gradient’s direction and changes only its magnitude; elementwise value clipping can change direction. Clipping entered common practice through recurrent-network training, where exploding gradients were routine, and remains a common Transformer ingredient. It is insurance, not a cure for a chronically excessive learning rate.

Two practical amendments complete the picture. First, computing the exact gradient means averaging over the whole training set — wasteful when a random minibatch of 32–512 examples gives an unbiased, if noisy, estimate for a fraction of the cost. This is stochastic gradient descent (SGD), rooted in Robbins and Monro’s 1951 work on stochastic approximation, and it is what everyone actually runs; “gradient descent” in deep learning almost always means minibatch SGD. Second, since the right learning rate changes as training proceeds, practitioners use schedules (typically decaying \(\eta\) over time), which we also defer to section 1.6.

The batch-size choice is a genuine trade, not a technicality. “Unbiased” means the minibatch gradient equals the full gradient on average over random batches (no systematic tilt, just noise), so many small noisy steps and few large exact steps both make progress, and the question is which buys more descent per FLOP. A batch of 32 computes its gradient at 1/1000th the cost of a 32,000-example dataset’s full gradient while its noise shrinks only with the square root of batch size: the small batch is noisier by a factor of ~30, but cheaper by a factor of 1000, and that trade overwhelmingly favors small batches when data is abundant. The practical batch size is then set less by statistics than by hardware (fill the accelerator’s parallelism, no more) and by a soft upper limit: past a data-dependent point, doubling the batch stops letting you double the learning rate, and the extra arithmetic buys almost nothing. One bookkeeping term to fix now, because every training log you read in this course uses it: an epoch is one full pass over the training data — at batch size 32 on 60,000 examples, 1,875 steps per epoch.

People. Léon Bottou spent years, beginning well before the deep learning boom, making the case that stochastic gradient methods, long viewed as crude, are precisely what large-scale learning needs: when data is abundant, fast noisy steps beat slow exact ones, because the real currency is accuracy per second of compute rather than accuracy per pass over the data. His analyses and his “SGD tricks” writings shaped how the field trains models to this day; the optimizers in section 1.6 are all descendants of this line of work.

Beyond the bowl: non-convexity

The quadratic bowl has one minimum, and gradient descent finds it from anywhere. Neural network losses do not return that favor: they are non-convex, with multiple minima, plateaus, and saddle points. The smallest possible demonstration is a double-well loss in one parameter — two valleys separated by a bump, with the left valley slightly deeper:

import numpy as np

np.random.seed(0)

# A non-convex loss: two valleys, the left one deeper
# L(w) = (w^2 - 1)^2 + 0.3*w;  dL/dw = 4*w*(w^2 - 1) + 0.3
def L(w):
    return (w ** 2 - 1) ** 2 + 0.3 * w

def dL(w):
    return 4 * w * (w ** 2 - 1) + 0.3

for w0 in [-1.5, 0.5, 1.5]:
    w = w0
    for _ in range(500):
        w -= 0.01 * dL(w)
    print(f"start w0 = {w0:>4.1f} -> converged to w = {w:.4f}, loss = {L(w):.4f}")

Running this prints:

start w0 = -1.5 -> converged to w = -1.0356, loss = -0.3054
start w0 =  0.5 -> converged to w = 0.9601, loss = 0.2941
start w0 =  1.5 -> converged to w = 0.9601, loss = 0.2941

Same loss, same learning rate, same number of steps — and the answer depends on where you start. Two of the three starting points roll into the shallower right valley and stay there, ending with a loss 0.6 higher than the run that started on the left. Gradient descent is a local method: it commits to the basin it starts in, and nothing in the update rule ever climbs a hill to check for something better on the other side.

For a 1980s pessimist this was a serious argument against neural networks: a million-parameter non-convex problem should be a million-dimensional minefield of bad local minima. Practice turned out gentler, for reasons that took until the 2010s to articulate. In very high dimensions, a critical point where every direction curves upward becomes combinatorially unlikely away from the bottom of the landscape — many zero-gradient points are saddle points, with descent directions available, rather than true traps (an argument developed empirically and theoretically by Dauphin and colleagues at NeurIPS 2014). Minibatch noise can help perturb an iterate away from some saddle regions, though that is only one part of modern optimization behavior. Empirically, distinct minima found from different seeds often score similarly on held-out data, but the seed-to-seed variation is still part of the result. Report multiple-seed uncertainty where runtime allows; when it does not, state the limitation rather than implying one run is a distribution.

Going deeper

Plain cross-entropy on one-hot labels says: push the true class’s probability toward 1, forever. Several widely used refinements amount to disagreeing with some part of “forever,” “1,” or “every example equally.”

Label smoothing softens the target itself. A common convention mixes the one-hot target with the uniform distribution: with smoothing parameter \(\varepsilon\) and \(C\) classes, the true-class target is \(1 - \varepsilon + \varepsilon/C\) and every other class receives \(\varepsilon/C\). Another convention assigns \(1 - \varepsilon\) to the true class and divides \(\varepsilon\) among the other \(C - 1\) classes; state which convention an API uses. The gradient consequence is that cross-entropy no longer drives the target all the way to a one-hot limit, which curbs the endless-confidence dynamic noted earlier, and in the Inception paper’s experiments improved ImageNet accuracy slightly. Label smoothing is now a routine ingredient in image-classification and machine-translation recipes.

Focal loss reweights examples instead of targets. Lin and colleagues (ICCV 2017) faced dense object detection, where easy background examples outnumber objects by orders of magnitude: each easy example contributes a tiny cross-entropy gradient, but millions of tiny gradients drown the signal from the rare hard ones. Their fix multiplies the loss by \((1 - p_{\text{true}})^\gamma\) (with \(\gamma = 2\) as the paper’s default), which leaves hard, misclassified examples (\(p_{\text{true}}\) small) nearly untouched while shrinking well-classified examples’ contribution by orders of magnitude. The same idea (down-weight what is already solved) transfers to tasks with extreme class imbalance. The blunter classical tool is class weighting: multiply each class’s loss term by, say, inverse class frequency, so the model cannot minimize the loss by ignoring a 1%-frequency class. If you face imbalance in your project, start with weighting (one line in nn.CrossEntropyLoss(weight=...)), and remember that aggregate accuracy is insufficient by itself — a 99%-negative dataset hands out 99% accuracy for predicting “negative” always.

Calibration asks whether the probabilities mean anything. A classifier is calibrated if, among predictions made with confidence 0.8, about 80% are correct. Minimizing cross-entropy on abundant data pushes toward calibration (it is a proper scoring rule: honest probabilities minimize it), but an overparameterized network that keeps training long after fitting its data drifts toward overconfidence. Here is both outcomes on a problem where the true noise level is known — same data, one model with 2 parameters and one with thousands:

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
np.random.seed(0)

# Ground truth is genuinely noisy: P(y=1 | x) = sigmoid(2x - 1)
def sample(n, rng):
    x = rng.uniform(-3, 3, size=(n, 1)).astype(np.float32)
    p = 1 / (1 + np.exp(-(2 * x - 1)))
    yl = (rng.uniform(size=p.shape) < p).astype(np.float32)
    return torch.from_numpy(x), torch.from_numpy(yl)

X_tr, y_tr = sample(200, np.random.default_rng(0))
X_te, y_te = sample(20000, np.random.default_rng(1))

def train(model, steps, lr=0.05):
    torch.manual_seed(0)
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.functional.binary_cross_entropy_with_logits(model(X_tr), y_tr)
        loss.backward()
        opt.step()
    return model

small = train(nn.Linear(1, 1), steps=2000)
big = train(nn.Sequential(nn.Linear(1, 64), nn.ReLU(),
                          nn.Linear(64, 64), nn.ReLU(),
                          nn.Linear(64, 1)), steps=6000)

for name, model in [("logistic regression", small), ("64-unit MLP, trained long", big)]:
    with torch.no_grad():
        p = torch.sigmoid(model(X_te)).numpy().ravel()
    yv = y_te.numpy().ravel()
    pred = p > 0.5
    acc = np.mean(pred == (yv > 0.5))
    conf = np.mean(np.where(pred, p, 1 - p))    # confidence in the predicted class
    print(f"{name:<26} test accuracy = {acc:.4f}   mean confidence = {conf:.4f}")

Running this prints:

logistic regression        test accuracy = 0.8847   mean confidence = 0.8755
64-unit MLP, trained long  test accuracy = 0.8651   mean confidence = 0.9427

The logistic model, which matches the true data-generating family, is nearly calibrated: 88.5% accurate, 87.6% average confidence. The overparameterized MLP, trained three times longer on the same 200 points, is both less accurate (86.5%) and more confident (94.3%): it has fit the noise in its small training set and reports certainty it has not earned, an eight-point confidence-accuracy gap. This is not a contrived failure. Guo and colleagues (ICML 2017) measured exactly this on modern architectures and found that networks of the 2016 era were systematically overconfident in ways 1990s-era networks were not, and that a comically simple fix, temperature scaling (divide all logits by one scalar \(T\) fitted on validation data after training), repairs most of the miscalibration without changing a single prediction’s argmax. If your model’s probabilities feed any downstream decision (medical triage, content filtering, anything with a threshold), calibration is not optional, and the fix costs one parameter.

The through-line of all four refinements: the loss function is an instrument, not a given. Cross-entropy on one-hot labels is the correct default, and this course uses it everywhere — but each term of it encodes a decision (target 1.0; every example weighted equally; probabilities taken at face value) that practice sometimes needs to revisit. When a later chapter hands you a loss with an unfamiliar extra factor, the right reading habit is the one this section practiced: differentiate it, look at which examples get a large gradient and which get silenced, and the design intent usually announces itself.

Common pitfalls

Symptom: loss becomes nan after a few steps or a few epochs. Cause: usually a numerically naive softmax/log, or a learning rate in the divergent regime (logits grow until exp overflows). Fix: use fused log-softmax losses (never log(softmax(z)) by hand), and try dropping lr by 10x — if nan disappears, it was divergence.
Symptom: classifier trains but plateaus at mediocre accuracy, loss curve strangely flat. Cause: double softmax — probabilities fed to a logits-expecting loss compress all gradients. Fix: pass raw logits to nn.CrossEntropyLoss (or your NumPy equivalent); apply softmax only for human-readable output.
Symptom: loss decreases but glacially, and the curve is nearly a straight shallow line. Cause: learning rate far too small (the 0.005 regime). Fix: raise lr by factors of 3–10 until the loss either drops fast or destabilizes, then back off one step.
Symptom: regression with MSE trains fine, but the same setup applied to 0/1 labels barely learns once predictions saturate. Cause: MSE through a sigmoid kills gradients on confident errors (the \(p(1-p)\) factor). Fix: cross-entropy is the correct default loss for classification.
Symptom: 98% accuracy that evaporates on real use. Cause: class imbalance — the model learned to predict the majority class, and accuracy rewarded it. Fix: report per-class metrics (precision/recall, confusion matrix), and reweight the loss or resample; consider focal loss for extreme cases.
Symptom: loss oscillates wildly step to step but trends down. Cause: minibatch noise plus a near-limit learning rate — often acceptable. Fix: judge by the smoothed trend; only intervene if the oscillation grows.

Try it yourself

In the MSE-vs-CE table, add a column for the absolute error loss \(L = \lvert p - y\rvert\) (gradient through the sigmoid). Does it fix MSE’s vanishing-gradient problem at \(z = -8\)? Why or why not?
In the outlier experiment, vary the corruption: 0, 4, 16, and 32 bad labels out of 100. Plot (or tabulate) each loss’s fitted intercept against corruption rate. Where does Huber start to break down?
Find the largest three-decimal learning rate for which the quadratic-bowl run still converges, and verify it approaches the theoretical bound \(2/25\). Then change the 25 to 100 and predict the new bound before running.
Write cross_entropy(logits, y) for a batch (2-D logits array, integer label vector) using the log-sum-exp trick, and test it on logits containing 1000.0. Compare against scipy.special.log_softmax or PyTorch if you have them available.
In the double-well experiment, add momentum: v = 0.9 * v - 0.01 * dL(w); w += v. Does the run starting at w0 = 1.5 now escape into the deeper left valley? Explain what momentum changed.
In the calibration experiment, create a separate validation set with a new random seed. Fit \(T\) on that validation set by minimizing cross-entropy, freeze it, and report calibration metrics on the untouched test set. How large is the fitted \(T\), and how much does test calibration improve?
Batch-size experiment: on the XOR dataset from section 1.1, compare full-batch gradient descent with minibatches of 10 at the same learning rate. Which reaches 100% accuracy in fewer examples processed?

Before class you should be able to

Explain the metric-versus-loss split and why accuracy itself gives gradient descent nothing to descend.
Choose among MSE, MAE, and Huber for a regression problem with outliers by reading their per-example gradients.
Explain why sigmoid-plus-MSE sends a vanishing gradient on confident errors, while cross-entropy’s gradient is \(p - y\).
Use softmax’s shift invariance to explain the stable log-softmax (the log-sum-exp trick), and say what the naive version does on a logit of 1000.
State what temperature does to a softmax distribution.
Recognize the three learning-rate regimes from a loss curve, and connect divergence to the curvature bound.
Keep the logits-versus-probabilities convention straight and spot a double-softmax bug from its symptoms.

Connects to

This session is covered by Quiz 1 (Module 1: S01–S08), given at S09.
The \(\operatorname{softmax} - \operatorname{onehot}\) gradient derived here becomes the first step of every backward pass in 1.4 (S04–S05).
The stable log-sum-exp cross-entropy reappears throughout the course, and in HW2’s training code; learning-rate schedules return in 1.6 (S06).

← S02 · From Linear Models to MLPs · All readings · S04 · Backpropagation I →

How to read this

Losses: one number to descend

Regression losses compared: MSE, MAE, Huber

Why not MSE for classification?

Softmax: from scores to probabilities

The overflow you will hit, and log-sum-exp

Gradient descent and the learning rate

Beyond the bowl: non-convexity

Going deeper

Common pitfalls

Try it yourself

Further reading

Before class you should be able to

Connects to