Module 2 · Reading 2.4–2.5 · read before S12: Training in Practice

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

How to read this

Part A — 2.4 The Training Recipe: Debugging Deep Networks

Most software fails loudly: a wrong index throws an exception, a bad type raises a TypeError. Neural network training fails silently. A model with a subtle bug (labels misaligned with inputs, a transform applied twice, a loss fed the wrong thing) usually still trains. The loss goes down, the accuracy goes up, nothing crashes; the model just ends up quietly worse than it should be, and nothing tells you. You cannot debug this by reading stack traces, because there are none.

What works instead is a discipline: a fixed sequence of cheap experiments, each designed so that a specific class of bug cannot hide from it. This section lays out that recipe, demonstrates its most powerful single test (overfitting one batch) catching a real bug live, and shows how to read training curves and choose a learning rate from evidence rather than folklore. Nothing here is deep theory. It is the difference between losing a week to a silent bug in HW2 and losing twenty minutes.

The recipe, in order

The steps build on each other; the expensive mistakes come from skipping ahead.

  1. Look at your data. Not statistics of it — actual examples, with their labels, after every preprocessing step. Plot a batch exactly as the model will see it. A large share of silent failures (wrong channel order, mangled normalization, off-by-one label alignment, augmentation destroying labels as in Section 2.3) are visible in thirty seconds of looking and nearly undetectable from loss curves.
  2. Establish dumb baselines. Predict the most common class; train a linear model on flattened pixels. These take minutes and calibrate every later number: if your CNN beats the class prior by two points, you have a bug or a hard task, and you need to know which.
  3. Get one end-to-end run working at toy scale. Tiny model, tiny data, a few minutes: input to loss to gradient step to evaluation number. Resist adding features (augmentation, schedulers, checkpointing) before this loop is trustworthy.
  4. Overfit a single batch. The keystone test, demonstrated below: your model and training loop must be able to memorize a handful of examples essentially perfectly. Until they can, nothing else is worth trying.
  5. Scale up until it hurts, then regularize. Grow data and model until you see genuine overfitting (defined by the curves below), then add regularization to trade training fit for validation performance.
  6. Tune, coarse to fine. Learning rate first (it usually matters more than any other single knob), then batch size, schedule, and regularization strength, changing one thing at a time and logging everything (Section 2.5).

Two meta-rules govern every step. Change one thing at a time: when a run improves after you altered the learning rate, the augmentation, and the model width together, you have learned nothing you can build on, and the time you “saved” by batching changes is repaid with interest during the confusion that follows. And keep the last working configuration sacred: before each modification, make sure you can return to a state you trust — a committed config, a saved seed, a checkpoint. Debugging deep networks is a search process, and a search needs a reliable way to backtrack. During the debugging phase specifically, fix the seed so runs are comparable; the multi-seed discipline of Section 2.5 is for measuring once things work, not for localizing bugs while they do not.

Historical note. Little of this appears in textbooks from the field’s first decades — it circulated as lab folklore, learned by apprenticeship. The best-known public codification is Andrej Karpathy’s 2019 blog post “A Recipe for Training Neural Networks,” which opens by arguing that neural net training tends to fail silently rather than loudly, and prescribes essentially the staged discipline above. The post became standard reading in industry teams and university courses alike; this section is the same philosophy, compressed and made runnable.

Overfit one batch first

The logic of the test: take a handful of examples (even with random labels) and train on nothing else. There is no generalization to worry about; a healthy modern network has vastly more than enough capacity to memorize 8 examples, so training loss must fall to approximately zero. If it does not, something between your input tensor and your optimizer step is broken, and you have localized the bug to that pipeline with a test that runs in seconds.

The demonstration below runs the test twice on a small CNN: once on a correct setup, and once with one of the most common real bugs in practice — applying a softmax before nn.CrossEntropyLoss, which already includes one internally.

import torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_num_threads(1)   # single-threaded: exact run-to-run reproducibility

def make_model(buggy=False):
    layers = [
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(16 * 4 * 4, 4),
    ]
    if buggy:
        layers.append(nn.Softmax(dim=1))   # BUG: CrossEntropyLoss wants logits
    return nn.Sequential(*layers)

# One batch of 8 random images with random labels -- nothing to generalize,
# everything to memorize. A healthy training setup must drive loss to ~0.
x = torch.randn(8, 1, 16, 16)
y = torch.randint(0, 4, (8,))

for buggy in [False, True]:
    torch.manual_seed(0)
    model = make_model(buggy)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    tag = "buggy (softmax before loss)" if buggy else "correct model"
    print(f"--- {tag} ---")
    for step in range(501):
        opt.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
        if step % 100 == 0:
            acc = (model(x).argmax(1) == y).float().mean().item()
            print(f"step {step:3d}: loss = {loss.item():.4f}, batch acc = {acc:.2f}")

Running this prints:

--- correct model ---
step   0: loss = 1.3411, batch acc = 0.75
step 100: loss = 0.0148, batch acc = 1.00
step 200: loss = 0.0024, batch acc = 1.00
step 300: loss = 0.0009, batch acc = 1.00
step 400: loss = 0.0004, batch acc = 1.00
step 500: loss = 0.0002, batch acc = 1.00
--- buggy (softmax before loss) ---
step   0: loss = 1.3746, batch acc = 0.75
step 100: loss = 0.7499, batch acc = 1.00
step 200: loss = 0.7448, batch acc = 1.00
step 300: loss = 0.7441, batch acc = 1.00
step 400: loss = 0.7439, batch acc = 1.00
step 500: loss = 0.7438, batch acc = 1.00

Three readings from real numbers. First, if initial predictions are roughly uniform over 4 classes, cross-entropy should sit near \(\ln(4) = 1.386\); both runs start near 1.34–1.37. A large departure is a diagnostic prompt to inspect class balance, bias initialization, output scale, and loss wiring — not proof of a bug by itself. Second, the correct model drives the batch loss to 0.0002. Third, the double-softmax model reaches batch accuracy 1.00 but stalls at loss 0.7438 because the extra softmax restricts what the logits-expecting loss receives.

Pitfall. The softmax-before-CrossEntropyLoss bug has a mirror image: applying no softmax where one is needed — for example, feeding raw logits to a metric or loss that expects probabilities, or using nn.NLLLoss (which expects log-probabilities) on logits. The convention to memorize: in PyTorch, classification models output raw logits, and nn.CrossEntropyLoss applies log_softmax internally. Add an explicit softmax only at inference time, and only if you actually need calibrated-looking probabilities rather than an argmax.

The test generalizes past this one bug. Labels re-shuffled at every step: can’t overfit (the model sees inconsistent targets, so the loss cannot fall). Labels misaligned once in the data pipeline pass: to this test they are one more random labeling, so step 1 of the recipe, looking at the data, must catch them. Every parameter frozen, or the output detached: the first backward() raises a RuntimeError, a loud failure. The model’s parameters missing from the optimizer: no weight changes, and the loss stays at its step-0 value. Only part of the network frozen or detached, such as a trunk under a trainable head: it can pass, because the trainable part still memorizes the batch (with the demonstration’s model and batch, a frozen trunk ends at 0.0379 against the correct run’s 0.0002). A missing zero_grad can pass too: with the demonstration’s model and batch, the loss swings up to 2.53, then still ends near 0. Model too small or activations saturating: can’t overfit. One cheap test, many bug classes with nowhere to hide.

The permuted-label test: capacity checks and leak detection

A second cheap experiment belongs next to overfit-one-batch in your toolkit: train on labels that have been permuted — each input keeps some label, but the assignment is scrambled once, at the source (before the pipeline copies or splits anything), so the labels are internally consistent and completely meaningless. Zhang, Bengio, Hardt, Recht, and Vinyals made this famous at ICLR 2017 by showing that standard ImageNet-scale architectures reach perfect training accuracy on fully randomized labels: memorization capacity is essentially never the binding constraint, and anything your model appears to “learn” could in principle be memorization. What makes the test operational is the validation side: a model trained on meaningless labels must score at chance on fresh validation data, because there is no signal to generalize. If it scores above chance, information about validation examples is reaching training (a leak), and you have detected it without inspecting a single line of the pipeline.

The script runs the test three times. First on a healthy pipeline with true labels (baseline), then on the same pipeline with permuted labels (the check should pass: memorized train set, chance validation), then on a leaky pipeline containing one realistic bug — the dataset is naively oversampled 2x before the train/validation split, so about half the validation examples end up with an identical twin (same input, same meaningless label) sitting in the training set:

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_num_threads(1)

def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)      # XOR-quadrant task
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def fit(Xtr, ytr, Xva, yva, steps=4000):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 256), nn.ReLU(),
                          nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 2))
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(steps):
        idx = torch.randint(0, len(Xtr), (64,))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        tr = (model(Xtr).argmax(1) == ytr).float().mean().item()
        va = (model(Xva).argmax(1) == yva).float().mean().item()
    return tr, va

rng = np.random.default_rng(0)

# Part A: the permuted-label sanity check on a healthy pipeline.
Xtr, ytr = make_data(64, rng)
Xva, yva = make_data(1000, rng)
y_perm = ytr[torch.randperm(len(ytr))]                # labels now meaningless

tr, va = fit(Xtr, ytr, Xva, yva)
print(f"true labels:     train acc = {tr:.4f}, fresh val acc = {va:.4f}")
tr, va = fit(Xtr, y_perm, Xva, yva)
print(f"permuted labels: train acc = {tr:.4f}, fresh val acc = {va:.4f}")

# Part B: the same check EXPOSES a leaky pipeline. Bug: naive 2x
# oversampling applied BEFORE the train/val split, so every example's
# duplicate (with its label) can land in the other split.
X, y = make_data(400, rng)
y_shuf = y[torch.randperm(400)]                       # permutation test labels
Xd, yd = torch.cat([X, X]), torch.cat([y_shuf, y_shuf])   # oversample first (BUG)
split = torch.randperm(800)                                # ...then split
tr_idx, va_idx = split[:400], split[400:]
tr, va = fit(Xd[tr_idx], yd[tr_idx], Xd[va_idx], yd[va_idx])
print(f"leaky pipeline, permuted labels: train acc = {tr:.4f}, "
      f"val acc = {va:.4f}  <- should be ~0.5!")

Running this prints:

true labels:     train acc = 1.0000, fresh val acc = 0.8340
permuted labels: train acc = 1.0000, fresh val acc = 0.5090
leaky pipeline, permuted labels: train acc = 0.8375, val acc = 0.6375  <- should be ~0.5!

Line by line. With true labels, the model fits the 64 training points perfectly and generalizes to 0.834; 64 training points hold it there: the labeling rule itself scores 0.873 on these 1,000 points. With permuted labels, training accuracy is still 1.0000: sixty-four arbitrary labelings, memorized outright, the Zhang result in miniature and a reminder that “my training accuracy is high” certifies nothing. Validation is 0.509: chance, exactly as meaningless labels demand. That pair of numbers is the healthy signature. Then the leaky pipeline: validation accuracy 0.6375 on labels that carry no information. Ten points above the 0.5375 that always answering this validation half’s majority class would score, conjured purely by duplicated examples straddling the split — the model memorized training twins and got their validation copies for free. In a real project this bug inflates every score you compute, with true labels too, and nothing else in the recipe would catch it: loss curves look normal, the single-batch test passes, accuracy is plausible. The permutation test is the cheapest leak detector there is; oversampling (split first, then oversample only the training rows), patient-level grouping, and time-series data (where random splits leak the future into the past) are the standard places it earns its keep, provided the labels are scrambled at the unit the leak copies: before oversampling, per patient rather than per image, and for a time series by shifting the whole label sequence rather than shuffling it (a shuffle destroys the very correlation that leaks).

Where does it slot into the recipe? Run it once when the pipeline first works end-to-end (step 3), and again any time the data plumbing changes — new splits, new preprocessing, merged datasets. It costs one training run at toy scale. The capacity half of the test (can the model memorize?) doubles as a stronger version of overfit-one-batch when you suspect the model, not the pipeline; the leak half is the only test in this section that inspects the data rather than the model, which is why no amount of model-side debugging substitutes for it.

Reading the curves: underfitting and overfitting

Once single-batch training works, scale up, and from then on, your primary instrument is the pair of curves: training loss and validation loss versus steps. Four patterns cover most of what you will see.

  • Both high, close together, still falling or flat: underfitting. The model has not extracted what the training data offers. Train longer, raise capacity, raise the learning rate, or reduce regularization.
  • Training low and falling, validation rising after a minimum: overfitting. The gap between curves is the model memorizing training idiosyncrasies. More data (or augmentation), more regularization, or a smaller model; early stopping takes the checkpoint at the validation minimum.
  • A persistent but stable gap with both curves flat: the generalization gap at equilibrium — normal, and small gaps are expected. Do not reflexively treat any gap as overfitting; the diagnostic sign is validation loss worsening while training loss improves.
  • Training loss near zero almost immediately, validation near chance: a training-only shortcut — suspect label information that reaches the model only in training (for example, a feature the training pipeline builds from the labels) rather than celebrating.

A subtle case worth knowing: validation loss can rise while validation accuracy still improves, because accuracy only checks the argmax while loss also punishes miscalibrated confidence. Track both, and decide in advance which one you select models on.

Some mechanics make the curves easier to read honestly. Plot loss on a log scale: the late-training regime where everything looks flat on a linear axis is where most decisions get made. Evaluate at fixed step intervals rather than epoch boundaries if you compare runs with different batch sizes, or the x-axes will not line up. Smooth the training curve (it is one noisy minibatch per point) but never the validation curve — smoothing can shift the apparent validation minimum by many evaluations, which moves your early-stopping decision. And log everything to files from the first run, not the tenth; the run you will most want to inspect is always the one you did not log (Section 2.5 makes this systematic).

Choosing the learning rate from evidence

No hyperparameter moves outcomes more. The failure modes are asymmetric: too low wastes compute and can look like underfitting; too high destroys training. Rather than guessing, measure. Smith’s learning-rate range test raises the learning rate at every step of one short run and reads the result against the rate (Smith’s paper plotted accuracy; library tools plot loss). The independent 60-step runs below are a related learning-rate sweep: easier to explain and compare, but not the exact one-run range-test algorithm used by tools such as lr_find.

import numpy as np
import torch
import torch.nn as nn

np.random.seed(0)
torch.manual_seed(0)

# A small but real task: classify which quadrant-pair a noisy blob is in.
def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)      # XOR-like: needs hidden layer
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def make_model():
    return nn.Sequential(nn.Linear(2, 32), nn.ReLU(),
                         nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2))

def train_sgd(lr, steps, log_every=None):
    rng = np.random.default_rng(0)
    torch.manual_seed(0)
    X, y = make_data(512, rng)
    model = make_model()
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    loss_fn = nn.CrossEntropyLoss()
    losses = []
    for step in range(steps):
        idx = torch.randint(0, len(X), (64,))
        opt.zero_grad()
        loss = loss_fn(model(X[idx]), y[idx])
        loss.backward()
        opt.step()
        losses.append(loss.item())
        if log_every and step % log_every == 0:
            print(f"  step {step:3d}: loss = {loss.item():.4f}")
    return losses

print("sane learning rate (SGD, lr = 0.1):")
train_sgd(0.1, 401, log_every=100)
print("too-high learning rate (SGD, lr = 10.0):")
train_sgd(10.0, 401, log_every=100)

# Learning-rate sweep: same budget (60 steps) at each candidate lr,
# report the mean loss over the last 10 steps.
print("lr sweep (60 SGD steps each):")
for lr in [1e-3, 1e-2, 1e-1, 3e-1, 1.0, 3.0, 10.0]:
    losses = train_sgd(lr, 60)
    tail = np.mean(losses[-10:])
    print(f"  lr = {lr:6.3f}: mean loss of last 10 steps = {tail:.4f}")

Running this prints:

sane learning rate (SGD, lr = 0.1):
  step   0: loss = 0.6935
  step 100: loss = 0.5455
  step 200: loss = 0.4205
  step 300: loss = 0.3239
  step 400: loss = 0.2923
too-high learning rate (SGD, lr = 10.0):
  step   0: loss = 0.6935
  step 100: loss = 0.9322
  step 200: loss = 4.8313
  step 300: loss = 2.5282
  step 400: loss = 2.2738
lr sweep (60 SGD steps each):
  lr =  0.001: mean loss of last 10 steps = 0.6883
  lr =  0.010: mean loss of last 10 steps = 0.6821
  lr =  0.100: mean loss of last 10 steps = 0.6306
  lr =  0.300: mean loss of last 10 steps = 0.5165
  lr =  1.000: mean loss of last 10 steps = 0.5471
  lr =  3.000: mean loss of last 10 steps = 0.7944
  lr = 10.000: mean loss of last 10 steps = 2.1409

At lr = 0.1, loss declines steadily from 0.6935 to 0.2923. At lr = 10.0, the same model on the same data jumps above 10 by step 2, then thrashes between about 0.7 and 5; over steps 3–400 its median is 2.28, more than three times its starting loss. Note it never NaNs: divergence often looks like this, a noisy plateau above initialization, which inexperienced readers misdiagnose as “the model isn’t learning” and treat by training longer. The sweep tells the story in one table: essentially no progress at 1e-3 and 1e-2 (too low for a 60-step budget), lowest at 0.3 on this seed, slightly worse at 1.0, deteriorating at 3.0, catastrophic at 10.0. The classic U-shape. Seeds 1–4 in place of the two 0s in train_sgd put the lowest at 1.0 every time, so the sweep finds the band (0.3 to 1), not the point: a defensible learning-rate choice from about two seconds of compute. On real workloads, tools like fastai’s lr_find automate the one-run range test that this sweep approximates; schedules (warmup, cosine decay, one-cycle) from Section 1.6 then vary the rate over training.

People. Leslie Smith, a research scientist at the U.S. Naval Research Laboratory, developed the learning-rate range test and cyclical learning rates (published at WACV 2017 after circulating from 2015) and later the related “1cycle” schedule — practical contributions from outside the field’s big industrial labs that became everyday tools, largely through their adoption in the fast.ai course and library.

Three refinements turn the sweep into a complete learning-rate policy. First, the answer depends on the optimizer: Adam’s per-parameter scaling means its usable band sits orders of magnitude below SGD’s (typically around 1e-3 versus 1e-1 for common vision setups; exercise 2 has you measure the shift on this exact task). A sweep is per-(model, data, optimizer); reusing one across optimizers is a category error. Second, the best constant rate is rarely the best policy: schedules that start high and decay (cosine annealing, step decay, or Smith’s one-cycle, which rises then falls) consistently beat constants in practice, because early training tolerates (and profits from) large steps while late training needs small ones to settle into a minimum. Third, warmup: a few hundred steps of linearly increasing rate at the very start protects against the near-random initial gradients taking destructive early steps. Warmup earns its keep especially with large batches and with architectures like Transformers (Chapter 3), where it is standard.

The learning rate is also entangled with batch size, and the entanglement has a usable first-order rule. A batch of 256 averages away more gradient noise than a batch of 32, so it supports a proportionally larger step: the linear scaling rule (multiply the batch size by \(k\), multiply the learning rate by \(k\)), which Goyal and colleagues (2017) used, together with a warmup phase, to train ImageNet ResNet-50 at batch size 8,192 in one hour on 256 GPUs while matching small-batch accuracy (the paper reports 23.74% top-1 error versus the 23.60% baseline). The rule is a heuristic with a stated domain of validity, not a law (the same paper documents it breaking down at still-larger batches), but for the factor-of-2-to-8 batch changes you will actually make (fitting a bigger batch after moving to a GPU, say), “scale the learning rate with the batch” is the right reflex, and “I changed the batch size but kept the learning rate” is a classic silent regression.

Exploding gradients, and the clip that catches them

Vanishing gradients (Section 2.2) have a louder sibling. When the per-layer factors of backpropagation are typically greater than one, gradients grow geometrically with depth — and unlike vanishing, which quietly stalls training, explosion destroys it: one oversized step flings the weights into a region where activations saturate or overflow, loss jumps, and within a few steps everything is NaN. Recurrent networks, which multiply by the same weight matrix at every time step, are the classic habitat (Pascanu, Mikolov, and Bengio’s ICML 2013 analysis of the problem is the standard reference), but any deep or badly initialized network can produce the same signature.

The standard mitigation is gradient clipping: after backward() and before step(), compute the global norm of all gradients, and if it exceeds a threshold, rescale the whole gradient vector down to that threshold. The direction of the update is preserved; only pathological magnitudes are capped. In PyTorch it is one line, torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm), and the demonstration below shows what it buys. The setup is engineered to explode (a deep tanh network with weights scaled 2x too large, plain SGD), and the two runs differ only in the clipping line:

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(1)

# A setup prone to exploding gradients: a deep tanh MLP with over-scaled
# initialization, plain SGD on a small regression task.
def make_model():
    torch.manual_seed(0)
    layers = []
    for i in range(8):
        layers += [nn.Linear(32 if i else 1, 32), nn.Tanh()]
    layers += [nn.Linear(32, 1)]
    net = nn.Sequential(*layers)
    with torch.no_grad():
        for p in net.parameters():
            if p.dim() == 2:
                p.mul_(2.0)                 # over-scaled init
    return net

rng = np.random.default_rng(0)
X = torch.from_numpy(rng.uniform(-2, 2, size=(256, 1)).astype(np.float32))
y = torch.sin(3 * X)

def train(clip):
    net = make_model()
    opt = torch.optim.SGD(net.parameters(), lr=0.3)
    loss_fn = nn.MSELoss()
    max_gnorm, first_bad = 0.0, None
    for step in range(801):
        idx = torch.randint(0, len(X), (32,))
        opt.zero_grad()
        loss = loss_fn(net(X[idx]), y[idx])
        loss.backward()
        gnorm = torch.nn.utils.clip_grad_norm_(net.parameters(),
                                               clip if clip else float("inf"))
        if torch.isfinite(gnorm):
            max_gnorm = max(max_gnorm, gnorm.item())
        elif first_bad is None:
            first_bad = step
        opt.step()
        if step % 200 == 0:
            with torch.no_grad():
                full = loss_fn(net(X), y).item()
            print(f"  step {step:3d}: full-data loss = {full:.4f}")
    print(f"  largest finite pre-clip grad norm: {max_gnorm:.2e}"
          + (f"; first non-finite gradient at step {first_bad}" if first_bad
             is not None else ""))

print("no clipping (SGD, lr = 0.3):")
train(clip=None)
print("clip_grad_norm_ to 1.0 (same everything else):")
train(clip=1.0)

Running this prints:

no clipping (SGD, lr = 0.3):
  step   0: full-data loss = 1.6183
  step 200: full-data loss = nan
  step 400: full-data loss = nan
  step 600: full-data loss = nan
  step 800: full-data loss = nan
  largest finite pre-clip grad norm: 4.00e+18; first non-finite gradient at step 19
clip_grad_norm_ to 1.0 (same everything else):
  step   0: full-data loss = 0.6865
  step 200: full-data loss = 0.0752
  step 400: full-data loss = 0.0671
  step 600: full-data loss = 0.0365
  step 800: full-data loss = 0.0482
  largest finite pre-clip grad norm: 9.32e+00

The unclipped run is dead by step 19: gradient norms reach 4e+18 (eighteen orders of magnitude beyond useful), and the first non-finite gradient arrives before the twentieth update, after which every loss is NaN forever. The clipped run, identical in every other respect, trains fine: loss falls from 0.69 to under 0.05, and the largest raw gradient it ever encountered was a tame 9.3 (the early monster gradients exist in this run too; clipping caps them at 1.0 before they touch the weights, and after the first few steps the trajectory stays in well-behaved territory). Two habits follow. Log the gradient norm every step — it is the earliest warning you get, spiking before the loss shows anything. And treat clipping as a seatbelt, not a cure: if your gradients need clipping constantly, the underlying problem (learning rate, initialization, normalization, data outliers) is still there, and the clip is hiding the diagnosis while capping your effective step size.

One vocabulary distinction, since both appear in codebases: norm clipping (used here) rescales the entire gradient vector when its global norm exceeds the threshold, preserving direction; value clipping (clip_grad_value_) caps each gradient component independently, which changes the update’s direction as well as its size. Norm clipping is the default recommendation and what the Pascanu analysis supports; value clipping survives mostly in older code and a few niches. Either way, the threshold is a hyperparameter — a common starting point is 1.0, but the principled move is to log unclipped norms for a few hundred healthy steps and set the threshold above their typical range, so the clip engages only on genuine outliers.

A field guide to failure smells

Quick diagnostics worth memorizing, each a symptom with a first hypothesis. Initial loss far from ln(num_classes) when predictions should be roughly uniform: inspect output scale, initialization, class balance, and loss wiring. Loss exactly flat from step 0 on a fixed batch (on fresh minibatches, no trend at the step-0 level, which looks like too low a rate): no weight is being updated, usually because the model’s parameters never reached the optimizer or opt.step() never runs; check the optimizer’s parameter list and compare one weight before and after a step (backward() still computes the gradients, so gradient norms do not point here; with the parameters missing from the optimizer they even grow step after step, because opt.zero_grad() no longer clears them). Freezing every parameter, or detaching the whole output or computing it under no_grad, does not give this symptom: the first backward() raises a RuntimeError. Doing the same to part of the network leaves the rest training, so the loss still falls. Loss NaN after some steps: test learning rate, unstable operations, and gradient norms. Accuracy stuck at the class prior: inspect the confusion matrix and label balance. Validation much better than training may reflect augmentation/dropout or leakage; train/validation good but test poor suggests distribution or pipeline shift.

Going deeper (Part A)

The recipe treats training as an engineering discipline; a few results complicate the tidy mental models behind it, and knowing them keeps you from over-trusting the folklore.

The overfitting story has an epicycle: double descent. The curves section implies a U: bigger models eventually overfit worse. Nakkiran and colleagues (ICLR 2020) documented, across architectures and datasets, a double descent pattern: as model size grows, test error falls, rises near the point where the model can just barely fit the training data (the interpolation threshold), then falls again as the model grows past it — and the same non-monotonicity can appear as a function of training epochs. In the heavily overparameterized regime where deep learning lives, “more capacity means more overfitting” is not reliably true. For the practical scales in this course the classical picture usually holds, but when a bigger model inexplicably generalizes better than a mid-sized one, you are not necessarily measuring a bug.

High accuracy does not mean trustworthy probabilities. The subtle case in the curves section (validation loss rising while accuracy improves) is one face of miscalibration. Guo, Pleiss, Sun, and Weinberger (ICML 2017) measured modern networks against their 1990s ancestors and report that the modern ones are systematically overconfident: their softmax scores overstate their probability of being correct, and the paper found a one-parameter fix (temperature scaling — divide the logits by a constant tuned on validation data) surprisingly effective. If your project’s model feeds a decision that cares about confidence and not just argmax, calibration is a separate property to measure, not a free byproduct of accuracy.

Why memorizing networks generalize at all is a genuinely open thread. The permuted-label experiment showed your model memorizing 64 arbitrary labels; Zhang et al.’s point at scale was that the same architecture, optimizer, and regularization that generalize on real labels will happily memorize random ones, so classical explanations of generalization that bound capacity (VC dimension, uniform convergence over the hypothesis class) cannot be the whole story. The working consensus points at implicit regularization: SGD’s noisy dynamics and the architecture’s structure bias training toward solutions that generalize, even though the hypothesis class contains memorizers. That is an active research area, not a settled account — which is worth remembering whenever a blog post explains generalization to you in one paragraph.

And the batch-size story continues past the linear rule. McCandlish, Kaplan, Amodei, and the OpenAI Dota team (2018) proposed measuring a gradient noise scale (roughly, the batch size at which averaging stops reducing gradient noise usefully) as a per-task predictor of the largest efficient batch, and found it varies by orders of magnitude across tasks, growing as models train. It is the beginning of an answer to “what batch size should this task use,” a question the linear scaling rule takes as given.

Common pitfalls (Part A)

  • Symptom: loss decreases but the single-batch test never reaches near-zero loss. Cause: a pipeline bug that caps expressiveness — double softmax, wrong loss/output pairing, targets misaligned with inputs. Fix: run the buggy-vs-correct comparison from this section; localize by simplifying (fewer layers, fewer examples) until the failure disappears.
  • Symptom: “the model isn’t learning” — loss noisy and flat at a high value. Cause: frequently a too-high learning rate thrashing, not a too-low one; the two look alike on a smoothed dashboard. Fix: run a learning-rate sweep; compare against the lr = 10.0 trajectory above before concluding you need more steps.
  • Symptom: validation loss rises while validation accuracy improves, and you stop training in panic. Cause: growing overconfidence on a minority of mistakes inflates loss without flipping argmaxes. Fix: track both; select on the metric you actually care about, and consider calibration separately.
  • Symptom: great numbers that later evaporate. Cause: evaluated on data the model effectively saw — duplicates across splits, augmentation leaking test images, tuning on the test set. Fix: audit the split before trusting any curve; run the permuted-label leak detector from this section; keep a final test set you touch once, per the protocol of Section 2.5.
  • Symptom: training runs cleanly for thousands of steps, then loss suddenly spikes or goes NaN. Cause: an exploding-gradient event — a rare batch, a scheduler raising the rate, or instability that was building silently. Fix: log gradient norms so the spike is visible before the crash; add clip_grad_norm_ as a seatbelt; if clipping engages constantly, lower the learning rate or fix the initialization instead of leaning on the clip.

Try it yourself (Part A)

  1. Extend the overfit-one-batch script with two more injected bugs: (a) shuffle y every step, (b) build the optimizer over another model’s parameters (torch.optim.Adam(make_model().parameters(), lr=1e-3)), and, in a separate run, freeze all parameters. Record what each does: a signature on the loss curve, or an error at the first backward().
  2. Reproduce the sweep with Adam instead of SGD. Where does the usable band sit now, and by how many decades did it move?
  3. Take the transfer-learning experiment from Section 2.3 and deliberately overfit it: train the from-scratch model to 100% training accuracy while logging validation accuracy each epoch. Identify the epoch where early stopping should have triggered.
  4. Write a ten-line sanity_check(model, loss_fn, x, y, num_classes) function that asserts (a) initial loss near ln(num_classes) and (b) loss below 0.01 after 300 Adam steps on one batch. Keep it; it is reusable in HW2 and your project.

Further reading (Part A)

  • A. Karpathy, “A Recipe for Training Neural Networks,” blog post, 2019. https://karpathy.github.io/2019/04/25/recipe/ — the codification of this section’s philosophy; read it in full before starting your project.
  • L. N. Smith, “Cyclical Learning Rates for Training Neural Networks,” WACV 2017. https://arxiv.org/abs/1506.01186 — the range test and cyclical schedules, with the reasoning behind sweep-and-look.
  • C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, “Understanding Deep Learning Requires Rethinking Generalization,” ICLR 2017. https://arxiv.org/abs/1611.03530 — networks fit random labels perfectly; the paper behind the permuted-label test and its uncomfortable implications.
  • P. Goyal et al., “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” arXiv 2017. https://arxiv.org/abs/1706.02677 — the linear scaling rule and warmup, with careful large-scale experiments on where they hold.
  • R. Pascanu, T. Mikolov, Y. Bengio, “On the Difficulty of Training Recurrent Neural Networks,” ICML 2013. https://arxiv.org/abs/1211.5063 — the standard analysis of exploding and vanishing gradients, and the norm-clipping remedy.
  • P. Nakkiran et al., “Deep Double Descent: Where Bigger Models and More Data Hurt,” ICLR 2020. https://arxiv.org/abs/1912.02292 — the non-monotonic test-error curves that complicate the classical overfitting story.
  • C. Guo, G. Pleiss, Y. Sun, K. Weinberger, “On Calibration of Modern Neural Networks,” ICML 2017. https://arxiv.org/abs/1706.04599 — modern networks are overconfident; temperature scaling as the one-parameter fix.
  • I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, Chapter 11: Practical Methodology. https://www.deeplearningbook.org/contents/guidelines.html — the textbook treatment of baselines, diagnostics, and when to gather more data.
  • Dive into Deep Learning, Chapter 19: Hyperparameter Optimization. https://d2l.ai/chapter_hyperparameter-optimization/ — systematic search methods for when one-knob-at-a-time stops being enough.
  • PyTorch torch.optim documentation, including learning-rate schedulers. https://pytorch.org/docs/stable/optim.html — the API for schedules, parameter groups, and clip_grad_norm_’s optimizer-adjacent machinery.

Part B — 2.5 Experiments and Reproducibility

Part A debugged a single training run; Part B is about making honest claims across runs — the same discipline pointed outward, at your experiments rather than your code.

Every claim you will make in this course (in HW write-ups, and above all in your semester project) has the same shape: setup A performs better than setup B. What makes that claim science rather than anecdote is that training is a randomized algorithm. Change nothing but the random seed and you get a different model with a different score; run A once and B once, and the difference you observe may be the seeds talking, not the setups. Deep learning has had public reckonings with exactly this — published comparisons that dissolved when someone reran them with more seeds or equal tuning effort.

This section is the course’s methods chapter. It shows, with real runs, how large seed-to-seed variance actually is; lays out the rules for fair comparisons — equal tuning budgets above all; and builds the mechanical habits (configs, structured logs, one-command reruns) that make your own results checkable, by you in the S27 week and by us if you are called for a code defense. It supports the project’s fair-comparison policy directly: you are graded on experimental design and analysis, not leaderboard position.

Where the randomness lives

A training run consumes randomness at every stage: weight initialization, the shuffling that assembles minibatches, augmentation sampling, dropout masks, and (the one people forget) the data split itself, if you carved validation out of training data randomly. Two consequences follow.

First, controlling the randomness gives exact repeatability: fix the seeds of every generator involved and the same code on the same machine produces the same floats. In PyTorch that means torch.manual_seed(...) plus seeding NumPy/Python if your pipeline uses them (a seeded np.random.default_rng(seed) object, as our examples use, is the cleanest form — it makes the dependence explicit and local rather than global).

Second, and more important scientifically: a seed is not a nuisance parameter, it is a sample. Each seed draws one model from the distribution that your setup (architecture, data, hyperparameters) defines. The honest object of study is that distribution, and any single run is one sample from it.

Keep two kinds of nondeterminism distinct in your head, because they need different tools. Algorithmic randomness (the initialization, shuffling, and augmentation draws above) is randomness your code requests, and seeds control it completely. Environmental nondeterminism (thread scheduling, floating-point reduction order, GPU kernel selection) is randomness the hardware and libraries introduce without being asked, and seeds do not touch it. The examples in this book are CPU-based and single-threaded partly to stay in the first regime, where “same seed, same floats” is a promise the platform actually keeps. When you move to GPUs for the project, you inherit the second regime, and the pitfall below becomes your problem.

Seeds and their limits, measured

Both points in one script. run_experiment(seed) wraps an entire training run (data generation, initialization, batching) under one seed, on the XOR-quadrant task from Section 2.4:

import numpy as np
import torch
import torch.nn as nn

# XOR-quadrant task: label = 1 if x and y have the same sign.
def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def run_experiment(seed):
    """One complete training run, fully controlled by `seed`."""
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(256, rng)
    Xte, yte = make_data(2000, rng)
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(),
                          nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2))
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for step in range(400):
        idx = torch.randint(0, len(Xtr), (64,))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        return (model(Xte).argmax(1) == yte).float().mean().item()

# Same seed twice: bitwise-identical result.
print(f"seed 0, run 1: {run_experiment(0):.4f}")
print(f"seed 0, run 2: {run_experiment(0):.4f}")

# Different seeds: same code, same data distribution, different answers.
accs = [run_experiment(s) for s in range(5)]
print("seeds 0-4:", ", ".join(f"{a:.4f}" for a in accs))
print(f"mean = {np.mean(accs):.4f}, std = {np.std(accs, ddof=1):.4f}, "
      f"spread = {max(accs) - min(accs):.4f}")

Running this prints:

seed 0, run 1: 0.8895
seed 0, run 2: 0.8895
seeds 0-4: 0.8895, 0.8720, 0.8670, 0.8865, 0.8670
mean = 0.8764, std = 0.0108, spread = 0.0225

Both promises delivered, with numbers. Seed 0 run twice gives 0.8895 twice: reproducibility works. And seeds 0 through 4 span 0.8670 to 0.8895: a 2.25-point accuracy spread from randomness alone, with nothing about the method changing. Now imagine comparing “my improvement” (one run, drew a lucky seed like 0) against “baseline” (one run, drew 0.8670): you would report a 2-point win that is pure noise. This is on a synthetic task; small real datasets are typically worse.

The defense is replication: run finalists under multiple seeds where runtime allows (3–5 is a useful course-scale target) and report mean and variation. If the budget cannot support that, state the limitation and avoid distributional claims. A mean difference small relative to uncertainty is not automatically a finding.

Historical note. The field learned this the hard way. Henderson et al.’s “Deep Reinforcement Learning that Matters” (AAAI 2018) showed that published RL baselines could change ranking based on random seeds and implementation details alone: among their demonstrations, averaging different subsets of seeds of the same algorithm produced curves that looked like different algorithms. In NLP, Melis, Dyer, and Blunsom (ICLR 2018) re-tuned a plain LSTM baseline with the same hyperparameter search effort as its newer competitors and found it matched or beat several architectures that had claimed to surpass it — the reported progress was partly a difference in tuning budgets. Papers like Dodge et al.’s “Show Your Work” (2019) turned the lesson into reporting norms: results as distributions over seeds and budgets, not single points.

Seeds have limits beyond variance, too. A fixed seed makes your specific result repeatable; it does not make it representative — fixing one seed and never varying it just hides the distribution. And exact bitwise reproducibility itself is fragile across environments: different library versions, different hardware, or GPU execution can all change results even with identical seeds.

Pitfall. “I set the seed and my two runs still differ” — on a GPU this is expected by default. Some CUDA/cuDNN kernels are nondeterministic (floating-point additions reordered across threads, algorithm autotuning), so seeded runs can diverge slightly; torch.use_deterministic_algorithms(True) plus related settings buys determinism at a speed cost; see PyTorch’s reproducibility notes. The CPU runs in this course are deterministic, which is one reason the homework autograders can match your numbers exactly. Record your library versions alongside your seeds; a seed only pins down randomness within one software environment.

How many seeds? The standard error of the mean

“Run 3–5 seeds” is the working rule, but it is worth knowing what those seeds buy and how the arithmetic scales. If per-seed scores behave like independent draws with standard deviation \(\sigma\), then the mean of \(n\) seeds has its own uncertainty — the standard error of the mean:

\[\mathrm{SEM} = \sigma / \sqrt{n}\]

which shrinks with the square root of \(n\): quadrupling the seeds halves the uncertainty of your reported mean. The script extends the seed study from 5 to 20 seeds and watches both estimates stabilize: the mean itself and the interval \(\text{mean} \pm 2\,\mathrm{SEM}\), a rough 95% confidence interval for where the true mean of the seed distribution sits:

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(1)

# Same task and training run as the seed study, now across 20 seeds.
def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def run_experiment(seed):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(256, rng)
    Xte, yte = make_data(2000, rng)
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(),
                          nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2))
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for step in range(400):
        idx = torch.randint(0, len(Xtr), (64,))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        return (model(Xte).argmax(1) == yte).float().mean().item()

accs = np.array([run_experiment(s) for s in range(20)])
print("accuracies, seeds 0-19 (sorted):")
print("  " + ", ".join(f"{a:.4f}" for a in np.sort(accs)))
print(f"mean = {accs.mean():.4f}, std = {accs.std(ddof=1):.4f}, "
      f"spread = {accs.max() - accs.min():.4f}")
for n in [3, 5, 10, 20]:
    sub = accs[:n]
    sem = sub.std(ddof=1) / np.sqrt(n)
    print(f"first {n:2d} seeds: mean = {sub.mean():.4f}, "
          f"SEM = {sem:.4f}, mean +/- 2*SEM = "
          f"[{sub.mean() - 2 * sem:.4f}, {sub.mean() + 2 * sem:.4f}]")

Running this prints:

accuracies, seeds 0-19 (sorted):
  0.8585, 0.8630, 0.8670, 0.8670, 0.8670, 0.8670, 0.8675, 0.8705, 0.8715, 0.8720, 0.8720, 0.8725, 0.8765, 0.8785, 0.8825, 0.8850, 0.8855, 0.8860, 0.8865, 0.8895
mean = 0.8743, std = 0.0089, spread = 0.0310
first  3 seeds: mean = 0.8762, SEM = 0.0068, mean +/- 2*SEM = [0.8625, 0.8898]
first  5 seeds: mean = 0.8764, SEM = 0.0048, mean +/- 2*SEM = [0.8667, 0.8861]
first 10 seeds: mean = 0.8733, SEM = 0.0030, mean +/- 2*SEM = [0.8674, 0.8792]
first 20 seeds: mean = 0.8743, SEM = 0.0020, mean +/- 2*SEM = [0.8703, 0.8783]

Three readings. The full 20-seed picture is wider than the 5-seed study suggested: the spread grew from 2.25 to 3.1 points (extreme seeds keep arriving as you sample more), and the sorted list shows the distribution’s shape — a clump in the high 0.86s with a tail toward 0.89. The SEM column behaves exactly as \(1/\sqrt{n}\) predicts, falling from 0.0068 at three seeds to 0.0020 at twenty. And the confidence intervals tell you what each budget can support: with 3 seeds, your mean is only pinned down to within about +/- 1.4 points, wider than many differences you might want to claim; with 20 seeds, to within +/- 0.4. Uncertainty in the mean is what matters for comparisons, and it is always smaller than the raw seed spread, but only by a factor you have to buy with runs.

Seeds compete with everything else for compute, so the practical question is allocation. The working pattern: during exploration, run broad and shallow (many configurations, one seed each), because at that stage you are hunting for large effects and discarding obviously bad regions, where single runs suffice. Once the field narrows to finalists, flip the ratio: few configurations, many seeds, because now the differences are small and only replication can resolve them. What wastes budget is the middle path (moderate seeds on moderate numbers of configurations), which is too noisy to rank finalists and too expensive to explore widely. Write the allocation into your project plan (“screening: 20 configs x 1 seed; confirmation: top 3 x 5 seeds”) so the two phases stay honest, and never promote a screening-phase number into a report — screening runs are for choosing, confirmation runs are for claiming.

Statistical power: what your seed budget can detect

The SEM tells you how well you know one condition’s mean. A comparison involves two, and the sharper question is: given my noise level and seed budget, what size of true difference can I even hope to detect? That is the statistical power of the experiment (the probability that a real effect of a given size produces a significant-looking result), and you can compute it for your own setup by simulation, no statistics tables required. The script below models each seed’s score as a draw from a normal distribution with sigma = 0.011 (about what the studies above measured), then plays out tens of thousands of A-vs-B comparisons. It first calibrates the detection threshold under the null hypothesis (no true difference), so that the false-positive rate is exactly 5%, then measures how often true gaps of 0.5, 1, and 2 sigma clear that threshold:

import numpy as np

rng = np.random.default_rng(0)

# Monte Carlo power analysis for A-vs-B comparisons.
# Model: each seed's score ~ N(mean, sigma^2), sigma from the 20-seed study.
sigma = 0.011          # per-seed std of test accuracy (measured above)

def detect(a, b, crit):
    """Detection statistic: |mean difference| / standard error of it."""
    se = np.sqrt(a.var(ddof=1, axis=1) / a.shape[1]
                 + b.var(ddof=1, axis=1) / b.shape[1])
    return np.abs(a.mean(axis=1) - b.mean(axis=1)) / se > crit

reps = 100_000
print(f"{'seeds':>5} {'crit':>5} " +
      " ".join(f"power(d={d:.3f})" for d in (0.005, 0.011, 0.022)))
for n in [3, 5, 10, 20]:
    # Calibrate the 5% false-positive threshold under the null (no difference),
    # instead of trusting small-sample formulas.
    a = rng.normal(0.0, sigma, size=(reps, n))
    b = rng.normal(0.0, sigma, size=(reps, n))
    se = np.sqrt(a.var(ddof=1, axis=1) / n + b.var(ddof=1, axis=1) / n)
    crit = np.quantile(np.abs(a.mean(axis=1) - b.mean(axis=1)) / se, 0.95)
    powers = []
    for delta in [0.005, 0.011, 0.022]:      # true gaps: 0.5, 1, 2 sigma
        a = rng.normal(0.0, sigma, size=(reps, n))
        b = rng.normal(delta, sigma, size=(reps, n))
        powers.append(detect(a, b, crit).mean())
    print(f"{n:5d} {crit:5.2f} " + " ".join(f"{p:13.3f}" for p in powers))

Running this prints:

seeds  crit power(d=0.005) power(d=0.011) power(d=0.022)
    3  2.80         0.071         0.158         0.458
    5  2.30         0.100         0.288         0.793
   10  2.11         0.160         0.557         0.988
   20  2.03         0.288         0.868         1.000

The table is sobering, and it should recalibrate how you read three-seed comparisons, your own and published ones. With 3 seeds per condition, a true 1-sigma gap (about a point of accuracy here) is detected only 15.8% of the time; even a substantial 2-sigma gap is missed more often than found (45.8% power). Five seeds detect the 2-sigma gap reliably (79.3%) but still mostly miss 1 sigma. Detecting a 1-sigma difference dependably takes on the order of 20 seeds per condition (86.8%), matching the classical rule of thumb that 80% power at a 5% false-positive rate needs roughly \(n = 16 \cdot (\sigma / \delta)^2\) runs per group. Two side lessons hide in the crit column. The calibrated thresholds (2.80 at n=3, drifting toward 2.03) are far above the 1.96 a large-sample formula would give — that is the small-sample penalty a naive z-test ignores, rediscovered here by brute force. And the whole analysis ran in under a second: once you have a \(\sigma\) estimate for your task, simulating your planned experiment before running it costs nothing and tells you whether the plan can possibly support the claim you hope to make. When power is low, the honest options are to increase seeds, reduce noise (bigger validation sets help), or study bigger effects, not to run the underpowered comparison and report whichever way it lands.

Pitfall. Low power does not just cause missed findings — it corrupts the findings you do make. If only large-looking differences clear your threshold, and your true effects are small, then the comparisons that do come out “significant” are disproportionately the ones where seed luck inflated the gap: the winner’s-curse effect, by which underpowered fields systematically overestimate effect sizes. If your three-seed ablation shows a shockingly large win, the correct emotional response is suspicion, and the correct next action is more seeds on exactly that comparison.

Fair comparisons: the equal-budget rule

Seed variance is the visible unfairness; tuning budget is the invisible one. If you spent two weeks tuning your proposed method and ran the baseline at default settings, your comparison measures effort, not merit — the Melis result above is exactly this error at publication scale. The project’s fair-comparison policy asks for the discipline the field converged on:

  • Equal tuning budgets. Give every method in a comparison the same search effort — same number of configurations tried, chosen the same way (grid, random, or a tuner), on the same validation data. Report the budget (“each method: 12 configs, 3 seeds each”).
  • Identical everything else. Same splits, same preprocessing, same augmentation, same evaluation code, same early-stopping rule. A comparison isolates one difference; each additional difference is a confound.
  • Tune on validation, report on test, touch test once. Every peek at test scores during development converts test data into validation data and inflates your final number.
  • Decide metrics in advance. Choosing the metric after seeing results is a multiple-comparisons trap; pick the primary metric when you design the experiment (your project proposal asks for exactly this).

A note on how to spend an equal budget: if each method gets twelve configurations, draw them by random search rather than a grid. Bergstra and Bengio’s JMLR 2012 study made the case that has held up since: with a fixed trial budget, random search explores each hyperparameter at many more distinct values than a grid does (a 3x4 grid tries only three learning rates; twelve random draws try twelve), which matters because a few hyperparameters usually dominate and you do not know in advance which. Random search is also embarrassingly parallel and extends gracefully: adding trials later does not invalidate the earlier ones, where refining a grid does. Log every trial, not just the winner — the shape of the whole search is part of your evidence, and “Show Your Work”-style reporting (Dodge et al., below) uses exactly that record to show how performance depends on budget.

Under these rules, “the simple baseline won” is a successful experiment — per the syllabus, a carefully designed project where the baseline wins earns its grade on design and analysis, which is the part this section trains.

Ablations: earning your explanations

A result tells you that your system works; an ablation tells you why, by removing or simplifying one component at a time and measuring the damage. If you claim augmentation, pretraining, and architecture each contribute, the evidence is runs without each. Good ablation practice inherits everything above: change exactly one thing per variant, keep budgets equal, and use the same set of seeds for every variant (pairing controls some noise — differences between variants are then not additionally confounded by different draws).

Pairing deserves one more sentence. Reusing a seed list guarantees matched data splits and, when designed explicitly, matched data order or augmentation streams. It does not make parameter initializations equivalent across architectures or across code paths that consume different numbers of random draws. Analyze per-seed differences only when the runs share a meaningful experimental unit; otherwise the same seed labels are bookkeeping, not automatic covariance. Record which random streams are intentionally coupled.

Here is a complete miniature: a three-variant ablation, three shared seeds per variant, driven by a config dictionary and logged to a JSON-lines file — the structure to copy for your project:

import json
import numpy as np
import torch
import torch.nn as nn

def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def build_model(width, depth):
    layers, d_in = [], 2
    for _ in range(depth):
        layers += [nn.Linear(d_in, width), nn.ReLU()]
        d_in = width
    layers.append(nn.Linear(d_in, 2))
    return nn.Sequential(*layers)

def run(config, seed):
    """Every choice lives in `config`; every run is a (config, seed) pair."""
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(256, rng)
    Xte, yte = make_data(2000, rng)
    model = build_model(config["width"], config["depth"])
    opt = torch.optim.Adam(model.parameters(), lr=config["lr"])
    loss_fn = nn.CrossEntropyLoss()
    for step in range(config["steps"]):        # equal budget for every variant
        idx = torch.randint(0, len(Xtr), (config["batch_size"],))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        acc = (model(Xte).argmax(1) == yte).float().mean().item()
    return {"config": config, "seed": seed, "test_acc": round(acc, 4)}

base = {"width": 32, "depth": 2, "lr": 1e-3, "steps": 400, "batch_size": 64}
variants = {
    "baseline":       dict(base),
    "half width":     dict(base, width=16),
    "half depth":     dict(base, depth=1),
}

log_path = "ablation_log.jsonl"
with open(log_path, "w") as f:
    for name, config in variants.items():
        accs = []
        for seed in range(3):                  # same 3 seeds for every variant
            record = run(config, seed) | {"variant": name}
            f.write(json.dumps(record) + "\n")
            accs.append(record["test_acc"])
        print(f"{name:15s} acc = {np.mean(accs):.4f} +/- {np.std(accs, ddof=1):.4f} "
              f"(seeds: {', '.join(f'{a:.3f}' for a in accs)})")

with open(log_path) as f:
    first = f.readline().strip()
print("first log line:")
print(first)

Running this prints:

baseline        acc = 0.8762 +/- 0.0118 (seeds: 0.889, 0.872, 0.867)
half width      acc = 0.8692 +/- 0.0098 (seeds: 0.880, 0.864, 0.863)
half depth      acc = 0.8570 +/- 0.0193 (seeds: 0.876, 0.857, 0.838)
first log line:
{"config": {"width": 32, "depth": 2, "lr": 0.001, "steps": 400, "batch_size": 64}, "seed": 0, "test_acc": 0.8895, "variant": "baseline"}

Read this like a referee. Halving the width costs 0.7 points on average (0.8762 to 0.8692), but the per-variant standard deviations are about a point, so with three seeds this difference is within noise, and an honest write-up says so. Halving the depth (one hidden layer instead of two) costs 1.9 points, more than the baseline’s seed standard deviation (1.2 points) but no more than its own (1.9): more suggestive, still worth more seeds before a strong claim. Notice how easily a single-seed version of this table would lie: on seed 0 alone, “half width” (0.880) and “half depth” (0.876) both look nearly indistinguishable from baseline (0.889). Averages with spreads, or no conclusions — and reporting an inconclusive ablation honestly (“the difference is within seed noise at our budget”) is itself a correct result. (The power analysis above applies verbatim here: with \(\sigma\) around a point and three seeds, only multi-point effects were ever likely to clear the noise, so the inconclusive width result is exactly what an underpowered probe of a small effect should produce.)

Ablation tables come in a few standard shapes, worth choosing among deliberately rather than by habit. Leave-one-out (the pattern above) starts from the full system and removes or simplifies one component per variant; it answers “what is each piece contributing, given everything else?” and is the default reviewers and graders expect. Additive build-up runs the other direction, starting from the barest baseline and adding one component at a time; the ConvNeXt paper discussed in Section 2.2 is a book-length example, and the shape is more informative when components were designed to work together, since it shows where diminishing returns set in. Sensitivity sweeps vary one continuous knob (width, augmentation strength) across several values, establishing that a choice sits on a plateau rather than a knife’s edge, a claim single-point ablations cannot make. Two cautions apply to all three shapes. Components interact: leave-one-out can report that neither A nor B matters when either alone suffices (removing A changes nothing because B covers for it), so when two components plausibly overlap, the honest table includes the remove-both variant. And each additional variant is another comparison drawn from the same noise, so a table with ten variants and three seeds each will contain at least one spurious-looking “effect” more often than not — pre-commit to which comparisons matter, and treat surprises in the margins as hypotheses, not findings.

People. David Picard, a researcher at École des Ponts ParisTech, made the variance point unignorable with a 2021 preprint titled “torch.manual_seed(3407) is all you need.” He trained the same small CIFAR-10 setups across roughly ten thousand seeds and found accuracy differences from seeds alone that rivaled gaps between published methods. The title is a joke about cherry-picking; the histograms show why a chosen seed count must be justified by budget and detectable effect rather than treated as a universal minimum.

Configs, logs, and the reproducible unit

The experimental habits above only work if the bookkeeping does. The pattern in the ablation script scales from homework to research lab:

  • One config object holds every choice. Model size, learning rate, steps, batch size, augmentation flags, data version — if it affects the outcome, it is in the config, not buried as a constant in code. A CPU run is then a pure function of (config, seed); ad-hoc edits between runs are how you end up unable to reproduce your own best number. Real projects graduate from dicts to YAML/JSON files or tools like Hydra, and to trackers like MLflow or Weights & Biases; the structure is identical.
  • Append one structured record per run. JSON lines (one JSON object per line) is greppable, plottable with three lines of pandas, and append-safe. Log the config, the seed, the metrics, and, in a git repository, the commit hash of the code that ran (git rev-parse HEAD), plus library versions.
  • Never overwrite results; never rely on memory. The log is the lab notebook. Your project report’s tables should be generated from it, not transcribed by hand — transcription is where numbers drift.
  • Make reruns one command. If reproducing a table requires undocumented steps, it is not reproducible — including by you, three weeks later, if you are called for a code defense.

This is also the cheapest insurance in the course: when a result looks surprising, the first debugging move is rerunning its exact (config, seed) from the log and confirming the number — one rerun separates “my code changed” from “I misremembered”.

A warning from every lab that has scaled this up: configs sprawl. Each experiment adds a flag; flags accumulate; within weeks the config has thirty fields, most combinations of which have never been run together and some of which silently conflict. Three habits keep sprawl survivable. Prune dead flags when a line of inquiry closes: a config field no experiment reads anymore is a trap for your future self. Never let code infer behavior from the absence of a key (defaults hidden in config.get(...) calls are where “I thought that run used augmentation” comes from); make defaults explicit in one place. And distinguish the experiment config (what you are varying) from the infrastructure config (paths, devices, worker counts) so that diffs between runs show only what matters. The tools mentioned above are largely mechanized versions of these habits: Hydra composes configs from files with explicit overrides; trackers like Weights & Biases or MLflow store the resolved config, metrics, and environment per run and give you the compare-two-runs diff for free. For a course project, the JSONL-plus-git-hash pattern in the script is genuinely enough — what matters is that some system, not your memory, is the source of truth.

Going deeper (Part B)

The methods in this section are the field’s working compromise, and the research frontier on “how should we run and report experiments” is livelier than you might expect.

Seed variance is only one slice of the variance. Bouthillier and colleagues (MLSys 2021) decomposed the run-to-run variability of standard benchmarks into its sources (weight initialization, data ordering, data splits, hyperparameter optimization) and report that the split and tuning stages contribute substantially; randomizing only the weight seed, as most papers do, understates total variance. Their recommendation is to randomize as many sources as possible and embrace slightly noisy estimates of a more honest quantity. The practical translation at course scale: vary the split along with the seed when your dataset is small, because a lucky split lies harder than a lucky initialization.

Small-sample reporting can be done much better than mean-plus-std. For settings where each run is expensive (a handful of runs is all anyone gets), Agarwal and colleagues (NeurIPS 2021) showed that common practices like reporting best runs or plain means over 3 seeds routinely reverse conclusions, and proposed sturdier summaries: interquartile means, performance profiles, and stratified bootstrap confidence intervals, packaged in their rliable library. The paper is about reinforcement learning benchmarks, but the statistics transfer to any few-runs regime — including a course project.

Power analysis has field-specific guides. Colas, Sigaud, and Oudeyer (2018) wrote a short practitioner’s tutorial on exactly the question our Monte Carlo answered (how many seeds a comparison needs), including the t-test and bootstrap machinery to do it with formulas instead of simulation. It is the natural next read if the power table surprised you.

And the institutions are adapting. NeurIPS ran a formal reproducibility program from 2019 — paper checklists asking authors to disclose seeds, budgets, and variance measures, plus a community challenge in which volunteers attempt to reproduce accepted papers; Pineau and colleagues’ report (JMLR 2021) documents what it changed. Preregistration (committing to hypotheses, metrics, and analysis before running, standard in medicine) exists in machine learning mostly as an ideal, but your project proposal’s experiment plan is a working approximation of it: metrics and budgets declared before results exist, which is precisely what makes the eventual claims checkable.

Common pitfalls (Part B)

  • Symptom: your “improvement” vanishes when someone reruns it. Cause: single-seed comparison inside the noise band, like the 2.25-point spread measured above. Fix: 3+ seeds per condition, report mean and spread, and claim only differences that are large relative to the spread.
  • Symptom: baseline loses to your method, but the baseline was run at defaults. Cause: unequal tuning budgets — the comparison measures your attention, not the methods. Fix: pre-commit an equal search budget per method and report it in the write-up.
  • Symptom: excellent validation-driven development, then a weak test number you are tempted to “fix” with one more test run. Cause: test-set reuse converting test into validation. Fix: freeze the final configuration on validation evidence, evaluate on test once, report whatever it says.
  • Symptom: you cannot recreate last week’s best result. Cause: untracked changes — an edited constant, a different data file, a lost seed. Fix: configs for every choice, JSONL logging with the git hash, and reruns by (config, seed) as in the ablation script.
  • Symptom: seeded runs match on your laptop but not on Colab. Cause: different library versions or GPU nondeterminism. Fix: record versions in the log, use deterministic-algorithms settings when it matters, and compare distributions across seeds rather than single floats across machines.

Try it yourself (Part B)

  1. Extend the seed study from 5 to 20 seeds and plot a histogram of test accuracies. How well did the 5-seed mean and std estimate the 20-seed distribution? What is the worst pairwise “improvement” you could have manufactured by picking one lucky and one unlucky seed?
  2. Add a fourth variant to the ablation (lr = 1e-2, everything else at baseline) with the same three seeds, appending to the same JSONL log. Then write a separate 10-line script that reads the log and regenerates the summary table — your first report-from-log pipeline.
  3. Take your Section 2.3 augmentation comparison and wreck it deliberately: give the augmented condition 3x the training steps. Observe how easily an unequal budget manufactures a “finding,” then state the equal-budget version of the claim.
  4. For your project idea, draft the experiment plan this section implies: the baseline and the method, the metric, how you will estimate uncertainty (your seed plan), a compute estimate, and the ablations you expect to need. This becomes a section of your proposal (due the week of S23).

Further reading (Part B)

  • P. Henderson et al., “Deep Reinforcement Learning that Matters,” AAAI 2018. https://arxiv.org/abs/1709.06560 — the paper that made seed sensitivity a mainstream concern, with rankings that flip on implementation details.
  • G. Melis, C. Dyer, P. Blunsom, “On the State of the Art of Evaluation in Neural Language Models,” ICLR 2018. https://arxiv.org/abs/1707.05589 — the equal-tuning-budget cautionary tale: a re-tuned LSTM baseline catching its published successors.
  • J. Dodge et al., “Show Your Work: Improved Reporting of Experimental Results,” EMNLP 2019. https://arxiv.org/abs/1909.03004 — reporting results as distributions over seeds and budgets, with expected-validation-performance curves.
  • D. Picard, “torch.manual_seed(3407) is all you need,” arXiv 2021. https://arxiv.org/abs/2109.08203 — ten thousand seeds, one architecture; the fastest cure for trusting single runs.
  • C. Colas, O. Sigaud, P.-Y. Oudeyer, “How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments,” arXiv 2018. https://arxiv.org/abs/1806.08295 — a practitioner’s guide to the seed-budget question, with the formula-based version of this section’s simulation.
  • X. Bouthillier et al., “Accounting for Variance in Machine Learning Benchmarks,” MLSys 2021. https://arxiv.org/abs/2103.03098 — decomposes benchmark variance into its sources; the case for randomizing more than the weight seed.
  • R. Agarwal et al., “Deep Reinforcement Learning at the Edge of the Statistical Precipice,” NeurIPS 2021. https://arxiv.org/abs/2108.13264 — robust few-run reporting: interquartile means and stratified bootstrap intervals, with an accompanying library.
  • J. Pineau et al., “Improving Reproducibility in Machine Learning Research,” JMLR 2021. https://arxiv.org/abs/2003.12206 — the NeurIPS reproducibility program report: checklists, challenges, and what moved the needle.
  • Hydra documentation (structured configs, composition, and command-line overrides). https://hydra.cc/ — the standard tool when the one-dict config pattern outgrows a single file.
  • PyTorch reproducibility notes. https://pytorch.org/docs/stable/notes/randomness.html — seeding, deterministic algorithms, and their costs, from the source.

Before class you should be able to

  • List the recipe’s six steps in order and state its two meta-rules.
  • Explain what the overfit-one-batch test proves, which bug classes it catches, and why initial loss should sit near ln(num_classes).
  • Read a training script and spot the double-softmax bug and its stuck-loss signature.
  • Use the permuted-label test to detect a data leak, and explain the oversample-before-split example.
  • Classify a pair of loss curves as underfitting, overfitting, a stable gap, or a training-only shortcut.
  • Run and read a learning-rate sweep, and recognize too-high-rate thrashing vs a too-low rate.
  • Explain why single-seed comparisons are untrustworthy and what to report instead (3+ seeds, mean and spread).
  • State the equal-budget rule and the touch-test-once rule.

Connects to

  • HW2 (CNN + transfer learning) goes out today — start at step 1 of this session’s recipe: look at make_synthetic_images’ output, then overfit one batch with your SmallCNN; the tests fix the learning rate.
  • This session is covered by Quiz 2 (Module 2: S09–S12), given at S13.
  • Section 2.5 is the methodological backbone of the project (pre-proposal, proposal, and report), and its habits are assumed in HW4 and examined in any code defense.

← S11 · Transfer Learning and Data Augmentation · All readings · S13 · RNNs and LSTMs →