Training in Practice: Debugging, Diagnosis, Experiment Tracking

Probe or fine-tune, measured: three ways to adapt the same trunk

How much of the trunk should train? Three answers on S11's miniature, at 24 and 200 target examples (reading 2.3): each strategy starts from the same pretrained trunk with a fresh head (model is the two in sequence).

# cond: one of "probe (frozen trunk)", "fine-tune (trunk lr/10)", "fine-tune (one big lr)"
if cond.startswith("probe"):                 # linear probe: trunk frozen
    for p in trunk.parameters():
        p.requires_grad = False
    groups = [{"params": head.parameters(), "lr": 1e-3}]
elif "lr/10" in cond:                        # discriminative learning rates
    groups = [{"params": head.parameters(), "lr": 1e-3},
              {"params": trunk.parameters(), "lr": 1e-4}]
else:                                        # one aggressive rate for everything
    groups = [{"params": model.parameters(), "lr": 1e-2}]
train(model, Xb, yb, epochs=150, param_groups=groups)   # this train takes param_groups
  

• This example's train takes param_groups in place of params and builds torch.optim.Adam(param_groups): a list of dictionaries, each naming some parameters and their learning rate.
• Discriminative learning rates: the head at the normal rate, the pretrained trunk at one tenth of it.

Freezing parameters does not freeze BatchNorm's statistics

Row 3 of the previous page, shown. A real backbone such as S11's ResNet-18 has a BN layer (S07) after every convolution; HW2's model has no BN. After import torch, import torch.nn as nn: nn.BatchNorm2d(8) normalizes each of 8 channels over \((N,H,W)\).

torch.manual_seed(0)
trunk = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.BatchNorm2d(8), nn.ReLU())
for p in trunk.parameters():
    p.requires_grad = False            # conv weights, BN's gamma and beta
bn = trunk[1]                          # one mean and variance per channel
print([name for name, _ in bn.named_buffers()])
x = torch.randn(32, 1, 16, 16) + 2.0   # a batch from a shifted domain
before = bn.running_mean.clone()       # a copy: BN updates running_mean in place
trunk(x)                               # a module starts in training mode
print(f"max change {(bn.running_mean - before).abs().max().item():.4f}")
  

['running_mean', 'running_var', 'num_batches_tracked']
max change 0.2110
  

named_buffers() prints BN's buffers: averages it updates on every forward pass, with no gradient, so requires_grad cannot stop them. In training mode BN moves its running statistics toward each batch's; in evaluation mode it uses them.

An outer model.train() switches a frozen trunk back

The previous page's natural fix is trunk.eval(). But a training loop calls model.train() at the start of every epoch, and train() and eval() reach every submodule. trunk, bn and x come from the previous page.

model = nn.Sequential(trunk, nn.Flatten(), nn.Linear(8 * 16 * 16, 2))
trunk.eval()                       # freeze BN's statistics, once, before training
model.train()                      # the first line of every epoch
before = bn.running_mean.clone()
model(x)                           # a training batch
print(f"trunk.training={trunk.training}, max change {(bn.running_mean - before).abs().max().item():.4f}")
  

trunk.training=True, max change 0.1899
  

model.train()                      # the next epoch starts
trunk.eval()                       # the fix: re-apply eval() after every model.train()
before = bn.running_mean.clone()
model(x)
print(f"trunk.training={trunk.training}, max change {(bn.running_mean - before).abs().max().item():.4f}")
  

trunk.training=False, max change 0.0000
  

Re-apply eval() to the frozen trunk after every model.train() (or override train()), and check running_mean before and after an epoch.

The test in code: a small CNN, 8 random images, 8 random labels

Reading 2.4's example, first half. The layers are HW2's SmallCNN(4); make_model(buggy=True) appends one more.

import torch
import torch.nn as nn
torch.manual_seed(0)
torch.set_num_threads(1)   # single-threaded: exact run-to-run reproducibility
def make_model(buggy=False):
    layers = [
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(16 * 4 * 4, 4),        # logits: (8, 4)
    ]
    if buggy:
        layers.append(nn.Softmax(dim=1))   # BUG: CrossEntropyLoss wants logits
    return nn.Sequential(*layers)
x = torch.randn(8, 1, 16, 16)      # 8 random images
y = torch.randint(0, 4, (8,))      # 8 random labels, 4 classes
  

nn.CrossEntropyLoss applies log_softmax to whatever it receives. The bug layer applies a softmax first, so the loss receives probabilities in \([0,1]\) where it expects logits.

One loop trains both models; only buggy differs

Second half: make_model, x and y from the previous page; 501 Adam steps on the one batch. argmax(1) picks each row's largest output, so acc is the share of the 8 labels predicted correctly.

for buggy in [False, True]:
    torch.manual_seed(0)                 # the same starting weights for both
    model = make_model(buggy)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    tag = "buggy (softmax before loss)" if buggy else "correct model"
    print(f"--- {tag} ---")
    for step in range(501):
        opt.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
        if step % 100 == 0:
            acc = (model(x).argmax(1) == y).float().mean().item()
            print(f"step {step:3d}: loss = {loss.item():.4f}, batch acc = {acc:.2f}")
  

The buggy model reaches 100% accuracy and still fails the test

The printout of the previous page's loop (reading 2.4):

--- correct model ---
step   0: loss = 1.3411, batch acc = 0.75
step 100: loss = 0.0148, batch acc = 1.00
step 200: loss = 0.0024, batch acc = 1.00
step 300: loss = 0.0009, batch acc = 1.00
step 400: loss = 0.0004, batch acc = 1.00
step 500: loss = 0.0002, batch acc = 1.00
--- buggy (softmax before loss) ---
step   0: loss = 1.3746, batch acc = 0.75
step 100: loss = 0.7499, batch acc = 1.00
step 200: loss = 0.7448, batch acc = 1.00
step 300: loss = 0.7441, batch acc = 1.00
step 400: loss = 0.7439, batch acc = 1.00
step 500: loss = 0.7438, batch acc = 1.00
    

The criterion is a loss near 0, not accuracy: accuracy only checks the argmax.

The task behind today's measurements: S10's XOR-quadrant miniature

Steps 4 and 5 are done (slides 11–19). Step 6 and everything after it measure something, so they need one task with real signal that trains in seconds: S10's XOR-quadrant data (reading 2.4–2.5). torch and nn as before.

import numpy as np
def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)
def build_model(width=32, depth=2):
    layers, d_in = [], 2
    for _ in range(depth):
        layers += [nn.Linear(d_in, width), nn.ReLU()]
        d_in = width
    layers.append(nn.Linear(d_in, 2))
    return nn.Sequential(*layers)
X, y = make_data(512, np.random.default_rng(0))
print(tuple(X.shape), y.float().mean().item())
      

(512, 2) 0.48828125

Two balanced classes, so the step-0 loss should sit near \(\ln2\approx0.693\).

Choose the learning rate from evidence: one short run per rate

Step 6 tunes the learning rate first: no other hyperparameter moves outcomes more. make_data, build_model from the previous page; plain SGD.

def train_sgd(lr, steps):
    rng = np.random.default_rng(0)
    torch.manual_seed(0)
    X, y = make_data(512, rng)
    model = build_model()
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    loss_fn = nn.CrossEntropyLoss()
    losses = []
    for step in range(steps):
        idx = torch.randint(0, len(X), (64,))
        opt.zero_grad()
        loss = loss_fn(model(X[idx]), y[idx])
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return losses
    

Each call is one complete, reproducible run that returns its minibatch loss at every step.

Too high a rate thrashes, mostly above its starting loss, without a NaN

train_sgd (previous page), 401 steps at 0.1 and at 10: the code prints every 100th loss, the figure plots all 401.

for lr in [0.1, 10.0]:
    losses = train_sgd(lr, 401)
    print(f"lr = {lr:4}:", ", ".join(f"{v:.4f}" for v in losses[::100]))
  

lr =  0.1: 0.6935, 0.5455, 0.4205, 0.3239, 0.2923
lr = 10.0: 0.6935, 0.9322, 4.8313, 2.5282, 2.2738
      

A loss that hovers above its step-0 value points first at a rate that is too high, not at too few steps.

The sweep: 60 steps per rate, lowest mean loss at 0.3

Seven independent train_sgd runs across four decades of rate, each scored by the mean loss of its last 10 steps.

for lr in [1e-3, 1e-2, 1e-1, 3e-1, 1.0, 3.0, 10.0]:
    losses = train_sgd(lr, 60)
    tail = np.mean(losses[-10:])
    print(f"  lr = {lr:6.3f}: mean loss of last 10 steps = {tail:.4f}")
  

  lr =  0.001: mean loss of last 10 steps = 0.6883
  lr =  0.010: mean loss of last 10 steps = 0.6821
  lr =  0.100: mean loss of last 10 steps = 0.6306
  lr =  0.300: mean loss of last 10 steps = 0.5165
  lr =  1.000: mean loss of last 10 steps = 0.5471
  lr =  3.000: mean loss of last 10 steps = 0.7944
  lr = 10.000: mean loss of last 10 steps = 2.1409
      

Lowest at 0.3 on this seed; seeds 1–4 put it at 1.0. A sweep finds the band (0.3 to 1), not the point. Each optimizer needs its own sweep: Adam's band usually sits near \(10^{-3}\).

The test in code: train, then report both accuracies

make_data, build_model from the task page, with the network 256 units wide so that it can memorize.

def fit(Xtr, ytr, Xva, yva, steps=4000):
    torch.manual_seed(0)
    model = build_model(width=256)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(steps):
        idx = torch.randint(0, len(Xtr), (64,))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        tr = (model(Xtr).argmax(1) == ytr).float().mean().item()
        va = (model(Xva).argmax(1) == yva).float().mean().item()
    return round(tr, 4), round(va, 4)          # (training acc, validation acc)
  

• Run it once the pipeline works end to end (step 3), and again whenever the data handling changes: new splits, new preprocessing, merged datasets. Split first, then oversample.

Overfit one batch checks the model and the loop; this test checks the split for leaks.

A leak, measured: 0.6375 on labels that carry no information

fit from the previous page; each line prints (training accuracy, validation accuracy).

torch.manual_seed(0)
rng = np.random.default_rng(0)
Xtr, ytr = make_data(64, rng)
Xva, yva = make_data(1000, rng)
y_perm = ytr[torch.randperm(64)]               # the same labels, scrambled once
print("true labels:     ", fit(Xtr, ytr, Xva, yva))
print("permuted labels: ", fit(Xtr, y_perm, Xva, yva))
X, y = make_data(400, rng)
y_shuf = y[torch.randperm(400)]                 # scrambled at the source, before copying
Xd, yd = torch.cat([X, X]), torch.cat([y_shuf, y_shuf])   # BUG: oversample 2x ...
idx = torch.randperm(800)                       # ... then split: copies land on both sides
tr, va = idx[:400], idx[400:]
print("leaky pipeline:  ", fit(Xd[tr], yd[tr], Xd[va], yd[va]))
  

true labels:      (1.0, 0.834)
permuted labels:  (1.0, 0.509)
leaky pipeline:   (0.8375, 0.6375)
  

Scrambled labels give validation 0.509, chance: the healthy signature. The leaky pipeline scores 0.6375 on labels with no information.

A seed is a sample: run the same experiment five times

The second half: once the pipeline is right, is a better number real? Debugging fixes one seed so that runs are comparable; measuring a result takes several. run_experiment is one complete run, data included, under one seed (make_data, build_model from the task page; Adam at its usual \(10^{-3}\), 400 steps).

def run_experiment(seed):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(256, rng)
    Xte, yte = make_data(2000, rng)
    model = build_model()
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for step in range(400):
        idx = torch.randint(0, len(Xtr), (64,))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        return (model(Xte).argmax(1) == yte).float().mean().item()
  

The function returns one test accuracy: one sample.

The same seed twice: identical. Five seeds: a 2.25-point spread

Call run_experiment (previous page) twice with seed 0, then for seeds 0 to 4. ddof=1 divides by \(n-1\).

print(f"seed 0, run 1: {run_experiment(0):.4f}")
print(f"seed 0, run 2: {run_experiment(0):.4f}")
accs = [run_experiment(s) for s in range(5)]
print("seeds 0-4:", ", ".join(f"{a:.4f}" for a in accs))
print(f"mean = {np.mean(accs):.4f}, std = {np.std(accs, ddof=1):.4f}, "
      f"spread = {max(accs) - min(accs):.4f}")
  

seed 0, run 1: 0.8895
seed 0, run 2: 0.8895
seeds 0-4: 0.8895, 0.8720, 0.8670, 0.8865, 0.8670
mean = 0.8764, std = 0.0108, spread = 0.0225
    

• On GPUs some CUDA/cuDNN kernels are nondeterministic even when seeded; library versions also change results, so record them with the seed.

Defense: 3–5 seeds per configuration, reporting the mean and the variation.

Ablations: every choice in a config, every run a (config, seed) pair

A result says that a system works; an ablation says why, by removing or simplifying one component at a time. Reading 2.5 moves each choice of run_experiment into a dictionary (make_data, build_model from the task page).

def run(config, seed):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(256, rng)
    Xte, yte = make_data(2000, rng)            # test: measured here, never used to choose
    model = build_model(config["width"], config["depth"])
    opt = torch.optim.Adam(model.parameters(), lr=config["lr"])
    loss_fn = nn.CrossEntropyLoss()
    for step in range(config["steps"]):        # equal budget for every variant
        idx = torch.randint(0, len(Xtr), (config["batch_size"],))
        opt.zero_grad()
        loss_fn(model(Xtr[idx]), ytr[idx]).backward()
        opt.step()
    with torch.no_grad():
        acc = (model(Xte).argmax(1) == yte).float().mean().item()
    return {"config": config, "seed": seed, "test_acc": round(acc, 4)}
  

The returned record, config plus seed plus metric, is what gets logged.

Three variants, the same three seeds, one JSON line per run

run from the previous page; JSON (JavaScript Object Notation) lines are written with json, in Python's standard library. Each variant changes one entry of base.

import json
base = {"width": 32, "depth": 2, "lr": 1e-3, "steps": 400, "batch_size": 64}
variants = {"baseline": dict(base), "half width": dict(base, width=16),
            "half depth": dict(base, depth=1)}  # a copy of base with depth replaced
with open("ablation_log.jsonl", "w") as f:
    for name, config in variants.items():
        accs = []
        for seed in range(3):                  # same 3 seeds for every variant
            record = run(config, seed) | {"variant": name}  # | adds a "variant" key
            f.write(json.dumps(record) + "\n")    # one line per run; "w" starts a new log
            accs.append(record["test_acc"])
        print(f"{name:15s} acc = {np.mean(accs):.4f} +/- {np.std(accs, ddof=1):.4f} "
              f"(seeds: {', '.join(f'{a:.3f}' for a in accs)})")
with open("ablation_log.jsonl") as f:
    print("first log line:")
    print(f.readline().strip())
  

Change one thing at a time, as when debugging: one entry per variant, the budget equal, the seeds reused.

Half width: within noise. Half depth: suggestive, not settled

The printout of the previous page (reading 2.5); +/- is the standard deviation over the 3 seeds.

baseline        acc = 0.8762 +/- 0.0118 (seeds: 0.889, 0.872, 0.867)
half width      acc = 0.8692 +/- 0.0098 (seeds: 0.880, 0.864, 0.863)
half depth      acc = 0.8570 +/- 0.0193 (seeds: 0.876, 0.857, 0.838)
first log line:
{"config": {"width": 32, "depth": 2, "lr": 0.001, "steps": 400, "batch_size": 64}, "seed": 0, "test_acc": 0.8895, "variant": "baseline"}
  

• Half width costs 0.7 points (0.8762 to 0.8692), but the standard deviations are about a point: within noise at 3 seeds, and an honest write-up says so.
• Half depth (one hidden layer instead of two) costs 1.9 points, above the baseline's standard deviation (1.2) but not its own (1.9): worth more seeds.
• On seed 0 alone the three look alike: 0.889, 0.880, 0.876.
• "The difference is within seed noise at our budget" is itself a correct result.

Averages with spreads, or no conclusions.