Module 2 · Reading 2.3 · read before S11: Transfer Learning & Data Augmentation

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

The architectures of Section 2.2 were trained on over a million labeled images. Your problems will not come with a million labels. A field-collected agriculture dataset, a medical imaging study, or a course project may have only hundreds to thousands of examples; a high-capacity CNN trained from scratch can then overfit easily unless the task, architecture, augmentation, and regularization are well matched.

Two techniques rescue the small-data regime, and they are the two workhorses of practical computer vision. Transfer learning reuses the features a network learned on a large task as the starting point for a small one, on the theory that edge, texture, and shape detectors are not task-specific. Data augmentation multiplies your labeled data by transforming it in label-preserving ways, encoding invariances you know the task has. This section demonstrates both with fully self-contained experiments (we will build a pretrained network on a synthetic task rather than downloading one, so every number is reproducible on your laptop in seconds) and closes with the ways each technique can quietly go wrong.

How to read this

Why features transfer at all

Section 2.1 ended with the observation that trained CNNs learn a hierarchy: first-layer kernels come out as oriented edges and color blobs, middle layers as textures and parts, late layers as task-specific configurations. The early and middle layers of that hierarchy are strikingly generic. Edges are edges whether the downstream task is retrievers-vs-terriers or healthy-vs-diseased leaves. So the trunk of a network trained on a big dataset is, in effect, a learned feature extractor for natural images in general, and the closer a layer sits to the input, the more general its features.

You can literally look at this hierarchy. Zeiler and Fergus (ECCV 2014) attached a “deconvolutional” visualizer to a trained AlexNet-style network and rendered, for units at each depth, the image patterns that excite them most: layer-1 units resolve to oriented edges and color opponencies, middle layers to textures, grids, and object parts, top layers to configurations tied to specific classes. The pictures made the generality gradient concrete, and they also served a debugging purpose in that paper, revealing pathologies in the first-layer filters that the authors fixed to improve on AlexNet’s benchmark result. The visualization habit is worth copying at your scale: the first-layer kernels of any conv net can be rendered as small images with no machinery at all, and eyeballing them tells you quickly whether pretraining produced structure or noise.

That intuition about where generality lives was established quantitatively the same year. Yosinski, Clune, Bengio, and Lipson transplanted layers between networks trained on disjoint halves of ImageNet and measured, layer by layer, where features stopped transferring — finding generality in early layers and increasing task-specificity with depth, plus a second, subtler cost from splitting co-adapted neighboring layers apart. The practical recipe crystallized around then and has barely changed since:

  1. Take a network pretrained on a large source task.
  2. Discard its classification head (task-specific by construction).
  3. Attach a fresh head for your task.
  4. Either freeze the trunk and train only the head, or fine-tune — continue training some or all trunk weights at a small learning rate.

Historical note. The speed of this shift is easy to underestimate. Through 2013–2014, papers like DeCAF (Donahue et al.) and “CNN Features off-the-shelf” (Razavian et al.) reported that features read out of a pretrained ImageNet network (AlexNet in DeCAF, OverFeat in Razavian et al.), used with nothing fancier than a linear classifier, beat carefully engineered state-of-the-art pipelines across a spread of recognition benchmarks. Within a couple of years, “download an ImageNet-pretrained backbone” replaced hand-designed features as the default starting point for essentially all of computer vision. The same pretrain-then-adapt pattern later became the organizing principle of NLP, as Chapter 3 covers.

Freeze or fine-tune? Freezing (the frozen trunk plus linear head is also called a linear probe) trains few parameters, so it resists overfitting, runs fast, and cannot damage the pretrained features; it is the right default when target data is scarce or closely resembles the source. Fine-tuning adapts the features themselves and wins when you have enough target data to support it or when the target domain differs (medical scans, spectrograms); it is standard to use a learning rate 10–100x smaller than for training from scratch, precisely to avoid destroying what pretraining built. A common middle path unfreezes only the last block or two (the most task-specific layers) while keeping early, generic layers fixed.

Transfer in miniature: a complete, self-contained experiment

Course rules for reading examples: no downloads. So instead of pulling an ImageNet backbone, we manufacture the entire situation in 16x16 synthetic images. Task A (the “big” source task, 2,000 examples): classify the orientation of a noisy bar as 0, 45, 90, or 135 degrees. Task B (the target task, 24 labeled examples): distinguish a “+” from an “x”. The tasks are related in exactly the way real transfer relies on: a “+” is a composition of 0- and 90-degree bars, an “x” of 45- and 135-degree bars, so orientation features learned on task A are the right vocabulary for task B. We compare a frozen transferred trunk against the identical architecture trained from scratch, across three seeds:

import numpy as np
import torch
import torch.nn as nn

# Task A (pretraining): classify the orientation of a noisy bar (4 classes).
# Task B (target):      classify "+" vs "x" -- compositions of those bars.
DIRS = {0: (0, 1), 1: (1, 1), 2: (1, 0), 3: (1, -1)}  # 0, 45, 90, 135 degrees

def draw_bar(img, r, c, d):
    dr, dc = DIRS[d]
    for t in range(-3, 4):
        rr, cc = r + t * dr, c + t * dc
        if 0 <= rr < 16 and 0 <= cc < 16:
            img[rr, cc] = 1.0

def make_task_a(n, rng):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 4, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        draw_bar(X[i], r, c, int(y[i]))
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_task_b(n, rng):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 2, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        if y[i] == 0:
            draw_bar(X[i], r, c, 0); draw_bar(X[i], r, c, 2)   # "+"
        else:
            draw_bar(X[i], r, c, 1); draw_bar(X[i], r, c, 3)   # "x"
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_trunk():
    return nn.Sequential(
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(),                       # -> 16 * 4 * 4 = 256 features
    )

def train(model, X, y, epochs, params=None):
    params = list(model.parameters()) if params is None else list(params)
    opt = torch.optim.Adam(params, lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(epochs):
        perm = torch.randperm(len(X))
        for i in range(0, len(X), 64):
            idx = perm[i:i + 64]
            opt.zero_grad()
            loss_fn(model(X[idx]), y[idx]).backward()
            opt.step()

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(1) == y).float().mean().item()

results = {"transfer": [], "scratch": []}
for seed in range(3):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xa, ya = make_task_a(2000, rng)             # task A: plenty of data
    Xb, yb = make_task_b(24, rng)               # task B: only 24 examples
    Xb_te, yb_te = make_task_b(500, rng)

    # 1) Pretrain trunk + 4-way head on task A.
    pre = nn.Sequential(make_trunk(), nn.Linear(256, 4))
    train(pre, Xa, ya, epochs=8)

    # 2) Transfer: keep the trunk, freeze it, train a fresh 2-way head.
    trunk = pre[0]
    for p in trunk.parameters():
        p.requires_grad = False
    torch.manual_seed(seed + 100)
    head = nn.Linear(256, 2)
    transfer = nn.Sequential(trunk, head)
    train(transfer, Xb, yb, epochs=150, params=head.parameters())

    # 3) Baseline: identical architecture, trained from scratch on task B.
    torch.manual_seed(seed + 100)
    scratch = nn.Sequential(make_trunk(), nn.Linear(256, 2))
    train(scratch, Xb, yb, epochs=150)

    results["transfer"].append(accuracy(transfer, Xb_te, yb_te))
    results["scratch"].append(accuracy(scratch, Xb_te, yb_te))
    print(f"seed {seed}: transfer = {results['transfer'][-1]:.4f}, "
          f"scratch = {results['scratch'][-1]:.4f}")

for name, accs in results.items():
    print(f"{name:8s} mean test accuracy = {np.mean(accs):.4f}")

Running this prints:

seed 0: transfer = 0.7640, scratch = 0.5940
seed 1: transfer = 0.7740, scratch = 0.5380
seed 2: transfer = 0.9480, scratch = 0.8800
transfer mean test accuracy = 0.8287
scratch  mean test accuracy = 0.6707

The transferred trunk wins in each of the three seeds, by 7 to 24 accuracy points, averaging 0.8287 versus 0.6707. Note what is and is not different between the two conditions: identical architecture, identical 24 training examples for task B, identical optimizer and budget. The only difference is where the trunk weights came from — random initialization versus task A. The from-scratch model reaches 100% training accuracy on the 24 examples (you can verify this; both models do), so its problem is pure overfitting: with 24 examples it cannot learn which of its randomly initialized features matter. The transferred model only has to learn a 2-class linear readout (514 trainable parameters) over features that already encode orientation. Also note the seed-to-seed spread: 0.76 to 0.95 within one condition. Small-data results are noisy, which is why this example reports three seeds and means; that discipline is the subject of Section 2.5.

The mechanics you just saw (requires_grad = False on trunk parameters, an optimizer over head.parameters() only, a fresh nn.Linear head) are what HW2 asks you to build: its SmallCNN is this architecture, and you write the freezing and the head-only training for this same experiment. With a real torchvision backbone in a project, the only things that change are the trunk (bigger) and the source task (ImageNet).

People. ImageNet, the dataset behind essentially all of this, was driven by Fei-Fei Li, then a new professor pursuing an unfashionable thesis: that data, not algorithms, was the bottleneck in computer vision. Built with her students (the paper, with Deng et al., appeared at CVPR 2009) using web-crawled images and crowdsourced labeling, it grew to over 14 million labeled images, and the associated ILSVRC competition (2010–2017) became the field’s proving ground — the stage on which AlexNet and ResNet had their moments in Section 2.2. Pretrained-on-ImageNet weights became infrastructure, arguably the dataset’s most lasting contribution.

Probe, fine-tune, or both: the choice, measured

The freeze-or-fine-tune decision from earlier deserves numbers of its own, because the folklore (“fine-tuning always wins if you can afford it”) hides a dependence on data size that you should see once. It also hides a mechanism worth stating before the code. At the start of adaptation, the new head is random, so the loss (and therefore every gradient flowing back into the trunk) is dominated by the head’s arbitrariness: the first few updates to pretrained weights are driven by noise. This is the argument for treating head and trunk differently: give the head a normal learning rate so it converges quickly, and give the trunk a small one (or zero, at first) so it cannot be damaged while the head is still nonsense. Discriminative learning rates implement the idea continuously; the probe-then-unfreeze schedule implements it in two discrete stages. The experiment extends the previous one with three adaptation strategies, each starting from the same pretrained trunk within each seed: a linear probe (trunk frozen); full fine-tuning with discriminative learning rates — the fresh head at the normal rate, the pretrained trunk at one tenth of it, the standard recipe for not bulldozing pretrained features while the randomly initialized head finds its feet; and full fine-tuning with one aggressive rate for everything. Each strategy runs at two target-data sizes, 24 and 200 labeled examples:

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(1)   # small model: thread overhead outweighs parallelism

# Linear probe vs full fine-tuning, at two target-data sizes.
# Same tasks as the transfer experiment: task A = bar orientation (source),
# task B = "+" vs "x" (target).
DIRS = {0: (0, 1), 1: (1, 1), 2: (1, 0), 3: (1, -1)}

def draw_bar(img, r, c, d):
    dr, dc = DIRS[d]
    for t in range(-3, 4):
        rr, cc = r + t * dr, c + t * dc
        if 0 <= rr < 16 and 0 <= cc < 16:
            img[rr, cc] = 1.0

def make_task_a(n, rng):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 4, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        draw_bar(X[i], r, c, int(y[i]))
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_task_b(n, rng):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 2, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        if y[i] == 0:
            draw_bar(X[i], r, c, 0); draw_bar(X[i], r, c, 2)
        else:
            draw_bar(X[i], r, c, 1); draw_bar(X[i], r, c, 3)
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_trunk():
    return nn.Sequential(
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(),
    )

def train(model, X, y, epochs, param_groups):
    opt = torch.optim.Adam(param_groups)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(epochs):
        perm = torch.randperm(len(X))
        for i in range(0, len(X), 64):
            idx = perm[i:i + 64]
            opt.zero_grad()
            loss_fn(model(X[idx]), y[idx]).backward()
            opt.step()

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(1) == y).float().mean().item()

conds = ["probe (frozen trunk)", "fine-tune (trunk lr/10)", "fine-tune (one big lr)"]
results = {(c, n): [] for c in conds for n in [24, 200]}
for seed in range(3):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xa, ya = make_task_a(2000, rng)
    pre = nn.Sequential(make_trunk(), nn.Linear(256, 4))
    train(pre, Xa, ya, epochs=8, param_groups=[{"params": pre.parameters(), "lr": 1e-3}])
    state = {k: v.clone() for k, v in pre[0].state_dict().items()}

    for n_target in [24, 200]:
        Xb, yb = make_task_b(n_target, rng)
        Xb_te, yb_te = make_task_b(500, rng)
        for cond in conds:
            trunk = make_trunk()
            trunk.load_state_dict(state)      # same pretrained weights each time
            torch.manual_seed(seed + 100)
            head = nn.Linear(256, 2)
            model = nn.Sequential(trunk, head)
            if cond.startswith("probe"):
                for p in trunk.parameters():
                    p.requires_grad = False
                groups = [{"params": head.parameters(), "lr": 1e-3}]
            elif "lr/10" in cond:             # discriminative learning rates
                groups = [{"params": head.parameters(), "lr": 1e-3},
                          {"params": trunk.parameters(), "lr": 1e-4}]
            else:                             # one aggressive lr for everything
                groups = [{"params": model.parameters(), "lr": 1e-2}]
            train(model, Xb, yb, epochs=150, param_groups=groups)
            results[(cond, n_target)].append(accuracy(model, Xb_te, yb_te))

for n_target in [24, 200]:
    print(f"--- {n_target} target examples ---")
    for cond in conds:
        accs = results[(cond, n_target)]
        print(f"{cond:24s} mean = {np.mean(accs):.4f} "
              f"(seeds: {', '.join(f'{a:.3f}' for a in accs)})")

Running this prints:

--- 24 target examples ---
probe (frozen trunk)     mean = 0.8287 (seeds: 0.764, 0.774, 0.948)
fine-tune (trunk lr/10)  mean = 0.8367 (seeds: 0.770, 0.798, 0.942)
fine-tune (one big lr)   mean = 0.8253 (seeds: 0.808, 0.724, 0.944)
--- 200 target examples ---
probe (frozen trunk)     mean = 0.9907 (seeds: 0.990, 0.988, 0.994)
fine-tune (trunk lr/10)  mean = 0.9953 (seeds: 0.994, 0.996, 0.996)
fine-tune (one big lr)   mean = 0.9947 (seeds: 0.990, 0.994, 1.000)

At 24 examples, the three strategies land at 0.825–0.837: differences of about a point, against per-condition seed spreads of 17 points or more. That is a tie: with this little data, the adaptation strategy is within noise, and the cheapest option (the probe) is the rational choice. At 200 examples the picture sharpens. Every strategy improves (more data helps everything), but now fine-tuning pulls measurably ahead: the probe’s mean error is 0.93%, the discriminative fine-tune’s 0.47%, halving the error rate consistently across seeds. Once the target data can support it, adapting the features beats reusing them as-is.

One honest caveat about the miniature. The one-big-lr condition was designed to show catastrophic feature destruction, and it does not: at 200 examples it performs as well as the careful recipe. The reason is scale — this trunk is small enough, and the task easy enough, that even scrambled features can be relearned from 200 examples inside the training budget. A 25-million-parameter backbone whose pretraining you cannot rerun does not forgive this way; the destruction failure mode in the pitfalls below is real at real scale, which is exactly why the discriminative-rate habit is worth building on a toy where you cannot yet see the damage.

Pitfall. Real backbones add a trap this miniature cannot show: normalization statistics. Freezing parameters via requires_grad = False does not freeze BatchNorm’s running mean and variance. Moreover, an outer model.train() call recursively puts the frozen trunk back into training mode even if you called trunk.eval() earlier. If pretrained statistics must stay fixed, reapply eval() to the frozen trunk/BN modules after each mode switch or override the model’s train() behavior; test the BN buffers before and after an epoch.

Data augmentation: a taxonomy

Augmentation attacks the same shortage from the other side: instead of importing knowledge through weights, inject it through the data. Each augmentation is a transformation you believe preserves the label; applying it randomly during training tells the network “these variations do not matter,” without collecting a single new example. Two ways to think about what that does. Statistically, you are replacing each training point with a cloud of nearby points sharing its label (the mixup paper frames this as training on a vicinity of the data rather than the data itself), which smooths the decision boundary in the directions the transforms span. Architecturally, augmentation is the soft cousin of the hard-wired invariances from Section 2.1: convolution builds translation structure into the weights; shift augmentation asks the network to learn the same structure from data; and for transformations with no architectural equivalent (lighting, occlusion), augmentation is the only channel available. The standard families:

  • Geometric: shifts, crops (random-resized crops are the ImageNet staple), horizontal flips, small rotations, scaling. Encode invariance to framing and pose.
  • Photometric: brightness, contrast, saturation, hue jitter; grayscale conversion; blur or noise. Encode invariance to lighting and sensor conditions.
  • Occlusion and mixing: random erasing/cutout blanks a patch (robustness to occlusion); mixup (Zhang et al., 2018) blends two images and their labels; CutMix pastes a labeled patch from one image into another. These regularize toward smoother decision boundaries.
  • Learned policies: AutoAugment and its cheaper successor RandAugment (Cubuk et al., 2020) tune the type and strength of transformations rather than hand-picking them.

The learned-policy family deserves a closer look, because it marks a shift in how the field thinks about augmentation: from a handful of hand-chosen transforms to a searchable hyperparameter. AutoAugment (Cubuk et al., CVPR 2019) framed the choice as a search problem (find, per dataset, a policy built from sub-policies like “rotate with probability p and magnitude m, then equalize” that maximizes validation accuracy) and ran a reinforcement-learning controller over thousands of candidate policies to find it. The search cost was enormous, and the follow-up made the pragmatic observation that most of the benefit survives radical simplification: RandAugment (Cubuk et al., NeurIPS 2020) throws away the learned structure, applies \(N\) transformations drawn uniformly from the standard set at one shared magnitude \(M\), and tunes just those two integers with a small grid search. Two knobs, competitive with the searched policies the paper compares against — a recurring pattern in deep learning, where an expensive learned solution reveals that a cheap randomized one was available all along. For your projects, the practical reading is: augmentation strength is a hyperparameter like the learning rate; the policy families give you a one-or-two-dial way to tune it instead of a dozen per-transform decisions.

Strength matters as much as type: augmentation strong enough to change semantics stops being regularization and becomes label noise. And every item on the list carries an implicit assumption about your task, which brings us to the failure mode.

Augmentation that helps, augmentation that lies

An augmentation is only valid if it preserves the label for your task. The same transformation can be correct on one dataset and catastrophic on another. The experiment below makes the point concretely. The task: distinguish an “L” shape from its mirror image, placed at random positions in a noisy 16x16 image, from only 32 training examples. Random shifts are label-preserving here (position never determines the class). Horizontal flips are label-destroying: flipping an L produces a mirrored L, so the flipped image belongs to the other class while keeping its old label.

import numpy as np
import torch
import torch.nn as nn

def draw_L(img, r, c, mirror=False):
    """An L shape (or its mirror image) with top corner at (r, c)."""
    col = c + 4 if mirror else c
    for t in range(7):
        img[r + t, col] = 1.0            # vertical stroke
    for t in range(5):
        img[r + 6, c + t] = 1.0          # horizontal foot
    return img

def make_data(n, rng):
    """Classify L (class 0) vs mirrored L (class 1) at random positions."""
    X = rng.normal(0.0, 0.15, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 2, size=n)
    for i in range(n):
        r = rng.integers(0, 9)
        c = rng.integers(0, 11)
        draw_L(X[i], r, c, mirror=bool(y[i]))
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_model():
    return nn.Sequential(
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(256, 2),
    )

def shift_aug(X, rng):
    """Random translation up to 3 px: label-PRESERVING for this task."""
    out = torch.empty_like(X)
    for i in range(len(X)):
        dr, dc = rng.integers(-3, 4, size=2)
        out[i] = torch.roll(X[i], shifts=(int(dr), int(dc)), dims=(1, 2))
    return out

def flip_aug(X, rng):
    """Random horizontal flip: label-DESTROYING for this task."""
    out = X.clone()
    mask = torch.from_numpy(rng.random(len(X)) < 0.5)
    out[mask] = torch.flip(out[mask], dims=[3])
    return out

def run(augment, seed):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    Xtr, ytr = make_data(32, rng)            # tiny training set
    Xte, yte = make_data(500, rng)
    model = make_model()
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(400):
        Xb = augment(Xtr, rng) if augment else Xtr
        opt.zero_grad()
        loss_fn(model(Xb), ytr).backward()
        opt.step()
    with torch.no_grad():
        return (model(Xte).argmax(1) == yte).float().mean().item()

for name, aug in [("no augmentation", None),
                  ("random shifts (label-preserving)", shift_aug),
                  ("horizontal flips (label-destroying)", flip_aug)]:
    accs = [run(aug, seed) for seed in range(3)]
    print(f"{name:37s} test acc = {np.mean(accs):.4f} "
          f"(seeds: {', '.join(f'{a:.3f}' for a in accs)})")

Running this prints:

no augmentation                       test acc = 0.7147 (seeds: 0.706, 0.634, 0.804)
random shifts (label-preserving)      test acc = 0.9633 (seeds: 1.000, 0.896, 0.994)
horizontal flips (label-destroying)   test acc = 0.5173 (seeds: 0.510, 0.462, 0.580)

The spread is the lesson. Shift augmentation lifts mean accuracy from 0.7147 to 0.9633: with 32 examples, telling the network “position does not matter” is worth more than any tuning you could do, because it is true for this task. Flip augmentation drops accuracy to 0.5173, statistically a coin flip: half the training signal now carries the wrong label, and the model cannot do better than ignore the very feature (chirality) that defines the classes. Same code, same effort — the difference is entirely whether the invariance you asserted actually holds.

Pitfall. Real datasets contain chirality and orientation semantics more often than people expect: handwritten characters (“b” flips to “d”, “6” rotates to “9”), traffic signs (turn-left vs turn-right), medical images with anatomical sidedness, text in photos. Horizontal flips are such a universal default for natural photos that they get copy-pasted into pipelines where they silently cap accuracy — the model trains without errors and converges; it just converges worse than it should. Audit your augmentation list against your label semantics before trusting any other diagnosis, and when in doubt, ablate: train once with and once without the suspect transform, as this section’s experiment does.

When transfer fails

Transfer is not a free lunch; it is a bet that the source features suit the target domain. The bet can lose. The failure has a standard name, domain shift (or covariate shift when you want to be precise that the input distribution \(p(x)\) moved while the labeling rule stayed put), and it is endemic in exactly the settings where transfer is most tempting: medical imaging (different scanners, different hospitals), agriculture (different fields, seasons, cameras), remote sensing (different sensors and altitudes). The cleanest way to see it is to hold everything fixed (tasks, architecture, budget) and shift only the target domain. The experiment below pretrains the usual orientation trunk on bright bars, then transfers it to two versions of the “+” vs “x” task: one in the matched domain (bright shapes on dark noise) and one with the contrast inverted (X = -X, dark shapes on bright noise). Semantically the tasks are identical — same shapes, same labels, same difficulty, and the from-scratch baseline confirms it. Only the low-level statistics changed:

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(1)   # small model: thread overhead outweighs parallelism

# When does transfer fail? Pretrain on bright bars (task A), then transfer to
# a target task whose domain either matches (bright "+" vs "x") or does not
# (contrast-INVERTED "+" vs "x": dark shapes on bright noise).
DIRS = {0: (0, 1), 1: (1, 1), 2: (1, 0), 3: (1, -1)}

def draw_bar(img, r, c, d):
    dr, dc = DIRS[d]
    for t in range(-3, 4):
        rr, cc = r + t * dr, c + t * dc
        if 0 <= rr < 16 and 0 <= cc < 16:
            img[rr, cc] = 1.0

def make_task_a(n, rng):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 4, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        draw_bar(X[i], r, c, int(y[i]))
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_task_b(n, rng, invert):
    X = rng.normal(0.0, 0.4, size=(n, 16, 16)).astype(np.float32)
    y = rng.integers(0, 2, size=n)
    for i in range(n):
        r, c = rng.integers(4, 12, size=2)
        if y[i] == 0:
            draw_bar(X[i], r, c, 0); draw_bar(X[i], r, c, 2)
        else:
            draw_bar(X[i], r, c, 1); draw_bar(X[i], r, c, 3)
    if invert:
        X = -X                     # same shapes, opposite contrast polarity
    return torch.from_numpy(X).unsqueeze(1), torch.from_numpy(y).long()

def make_trunk():
    return nn.Sequential(
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(),
    )

def train(model, X, y, epochs, params=None):
    params = list(model.parameters()) if params is None else list(params)
    opt = torch.optim.Adam(params, lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(epochs):
        perm = torch.randperm(len(X))
        for i in range(0, len(X), 64):
            idx = perm[i:i + 64]
            opt.zero_grad()
            loss_fn(model(X[idx]), y[idx]).backward()
            opt.step()

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(1) == y).float().mean().item()

for invert in [False, True]:
    tag = "shifted domain (inverted)" if invert else "matched domain"
    frozen_accs, scratch_accs = [], []
    for seed in range(3):
        rng = np.random.default_rng(seed)
        torch.manual_seed(seed)
        Xa, ya = make_task_a(2000, rng)
        Xb, yb = make_task_b(24, rng, invert)
        Xb_te, yb_te = make_task_b(500, rng, invert)

        pre = nn.Sequential(make_trunk(), nn.Linear(256, 4))
        train(pre, Xa, ya, epochs=8)
        trunk = pre[0]
        for p in trunk.parameters():
            p.requires_grad = False
        torch.manual_seed(seed + 100)
        head = nn.Linear(256, 2)
        transfer = nn.Sequential(trunk, head)
        train(transfer, Xb, yb, epochs=150, params=head.parameters())
        frozen_accs.append(accuracy(transfer, Xb_te, yb_te))

        torch.manual_seed(seed + 100)
        scratch = nn.Sequential(make_trunk(), nn.Linear(256, 2))
        train(scratch, Xb, yb, epochs=150)
        scratch_accs.append(accuracy(scratch, Xb_te, yb_te))

    print(f"{tag:26s} frozen transfer = {np.mean(frozen_accs):.4f}, "
          f"scratch = {np.mean(scratch_accs):.4f}")

Running this prints:

matched domain             frozen transfer = 0.8287, scratch = 0.6707
shifted domain (inverted)  frozen transfer = 0.6347, scratch = 0.6627

In the matched domain, the familiar 16-point transfer win (0.8287 vs 0.6707). In the shifted domain, the win is gone: the frozen trunk (0.6347) now slightly trails training from scratch (0.6627). The trunk’s first-layer kernels learned to fire on bright-bar-on-dark-noise patterns; fed the negation, they respond to a pattern of activations the downstream layers never saw in pretraining, so the inverted responses are harder to read, and 24 examples are not enough for the head to learn to read them: trained on 500 inverted task-B images instead of 24, the same frozen trunk’s linear head reaches 0.83–0.90 test accuracy across the three seeds. The scratch model, indifferent to polarity conventions it never had, is unaffected (0.6707 vs 0.6627 across domains; the task is equally hard either way). Had the frozen trunk lost to random initialization in every seed, transfer would be negative: the prior worse than no prior. Here it is no better than training from scratch: 0.635 against 0.663 on average, and the three seeds disagree.

The full-scale version of this finding caused a stir. Raghu, Zhang, Kleinberg, and Bengio’s “Transfusion” study (NeurIPS 2019) examined the near-universal practice of initializing medical-imaging models from ImageNet weights and reported that, on the two large-scale medical tasks they studied, transfer offered little performance benefit over training from scratch: small, simple architectures did about as well as giant pretrained ones, and much of what did transfer was low-level statistics rather than high-level features. Retinal scans and X-rays sit far from ImageNet’s photographs, in exactly the way our inverted images sit far from the pretraining bars. Related boundary results: He, Girshick, and Dollár (“Rethinking ImageNet Pre-training,” ICCV 2019) report that on the COCO detection benchmark, models trained from scratch match their pretrained counterparts if trained long enough — pretraining bought speed, not final accuracy; and Kornblith, Shlens, and Le (CVPR 2019) found that while better ImageNet models generally transfer better across a 12-dataset suite, the correlation weakens on fine-grained tasks whose distinctions ImageNet never needed. None of this demotes transfer from its default status (with scarce data it remains the single highest-leverage decision), but it replaces “always start from ImageNet” with a calibrated version: expect transfer to pay in proportion to domain similarity and data scarcity, and when your domain is exotic, measure the scratch baseline instead of assuming it loses.

Putting the two together

In practice transfer and augmentation are defaults, not options, and they compose: a typical small-data pipeline fine-tunes a pretrained backbone with task-appropriate augmentation on the target data. Their combined effect is why small-data vision is feasible at all. For a concrete decision procedure for a project built on a real pretrained backbone, work down this list: match the source model’s preprocessing exactly (input size, normalization constants); start with a linear probe to get a fast, hard-to-break baseline; add the augmentations whose invariances you have checked against your labels; then, if validation accuracy has room and you have more than a few dozen examples per class, unfreeze — last block first, discriminative learning rates, watching for the collapse signature from the pitfalls below. Each step is cheap to try and easy to undo, which is what you want in the small-data regime where every evaluation is noisy (Section 2.5).

Three closing rules of thumb. First, the scarcer your labels, the more freezing and suitable augmentation may earn. Second, tune augmentation strength on validation data: even with abundant data, augmentation can improve generalization, although excessive transforms can slow convergence or shift the training distribution. Third, validation/test pipelines normally disable stochastic augmentation while retaining the deterministic resizing, tensor conversion, and normalization required by the model. Test-time augmentation is a separate, explicit choice.

Going deeper

The pretrain-then-adapt recipe you practiced here is the small end of what is now the organizing pattern of the entire field, and several refinements are worth knowing as you read current work.

Pretraining no longer requires labels. Everything above assumed a supervised source task. Self-supervised methods manufacture the source task from raw images instead: contrastive approaches like MoCo (He et al., CVPR 2020) and SimCLR (Chen et al., ICML 2020) train the network to map two augmentations of the same image close together and different images apart (note augmentation reappearing, now as the definition of the pretraining task rather than a regularizer), while masked autoencoders (He et al., CVPR 2022) hide most of an image’s patches and train the network to reconstruct them. Both families report linear-probe and fine-tuning results competitive with supervised pretraining. The practical consequence is large: any domain with abundant unlabeled images (satellite tiles, pathology slides, field-camera streams) can now grow its own trunk.

What actually transfers is partly mundane. Neyshabur, Sedghi, and Zhang (“What is being transferred in transfer learning?”, NeurIPS 2020) dissected fine-tuned networks and found that a substantial share of transfer’s benefit comes from low-level statistics and from optimization effects — pretrained models land in flatter, better-behaved regions of the loss surface, and separately fine-tuned copies stay in the same basin, while from-scratch runs scatter. This aligns with the Transfusion findings above and cautions against romantic readings of “the network transfers its understanding of edges and objects.”

The order of adaptation matters at distribution boundaries. Kumar and colleagues (ICLR 2022) analyze a subtlety our miniature is too small to show: full fine-tuning can distort pretrained features in a way that helps in-distribution accuracy but hurts on shifted test data, and the simple fix (linear-probe first to settle the head, then fine-tune everything) reportedly captures the best of both. The two-stage recipe costs one extra training phase and is a sensible default when you care about robustness.

And scaled up, the recipe got a name. Models like CLIP (Radford et al., ICML 2021), pretrained on hundreds of millions of web image-text pairs and transferring to new tasks zero-shot via text prompts, extend “pretrain big, adapt cheap” past anything requiring your 24 labels at all. The term foundation model was coined for this regime, and Chapter 3 treats its language-side versions, where adapters and LoRA replace the full fine-tuning you did here for the same reason discriminative learning rates exist: touching all the weights of a huge pretrained model is expensive and risky, so touch as few as possible.

Common pitfalls

  • Symptom: fine-tuning starts well, then accuracy collapses below the frozen baseline. Cause: learning rate appropriate for a random init is far too large for pretrained weights — it destroys the transferred features in a few steps. Fix: fine-tune at 10–100x lower learning rate, or freeze the trunk first, train the head to convergence, then unfreeze gradually.
  • Symptom: frozen-trunk model trains but performs near chance. Cause: the head was attached to the wrong tensor (for example, logits of the old task instead of the features before the head), or preprocessing does not match what the trunk was trained on (wrong normalization or input size). Fix: check the feature dimension flowing into the new head and replicate the source model’s preprocessing exactly.
  • Symptom: augmented training loss stays high and validation is no better than without augmentation. Cause: augmentation may be too strong, or stochastic transforms may have leaked into validation. Fix: visualize augmented batches, dial strength down, and confirm that evaluation keeps required deterministic preprocessing but disables random augmentation.
  • Symptom: model performs well except on one specific class pair. Cause: a label-destroying augmentation, as in the flip experiment — the transform maps one class into another. Fix: remove or restrict the transform; verify with a two-run ablation.
  • Symptom: the pretrained backbone gives little or no advantage over training from scratch on your data. Cause: domain mismatch — your images’ low-level statistics (polarity, modality, texture) sit far from the source domain, as in the inverted-contrast experiment. Fix: verify preprocessing matches the source model first; then try unfreezing earlier layers so low-level features can adapt, and take the scratch baseline seriously rather than assuming transfer must win.

Try it yourself

  1. Add a third condition to the transfer experiment: full fine-tuning — initialize from the pretrained trunk but leave every parameter trainable on task B’s 24 examples. Where does it land relative to frozen and scratch, and does the answer change with 200 task-B examples?
  2. Make tasks A and B less related: pretrain on bars that only ever appear in the top-left quadrant, or on horizontal/vertical bars only (dropping the diagonals), and rerun. How much of the transfer advantage survives?
  3. In the augmentation experiment, add vertical flips and 90-degree rotations. Predict, before running, whether each is label-preserving for the L task; then check.
  4. Combine both: transfer the task-A trunk to the L-vs-mirror-L task with shift augmentation. Do the two techniques stack, or does one subsume the other here?

Further reading

  • J. Yosinski, J. Clune, Y. Bengio, H. Lipson, “How transferable are features in deep neural networks?”, NeurIPS 2014. https://arxiv.org/abs/1411.1792 — the layer-transplant experiments that established “general early, specific late,” the empirical backbone of this section.
  • J. Donahue et al., “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition,” ICML 2014. https://arxiv.org/abs/1310.1531 — early evidence that off-the-shelf CNN features beat engineered pipelines across benchmarks.
  • A. S. Razavian et al., “CNN Features off-the-shelf: an Astounding Baseline for Recognition,” CVPR Workshops 2014. https://arxiv.org/abs/1403.6382 — the companion result: a linear classifier on OverFeat features as a shockingly strong default.
  • S. Kornblith, J. Shlens, Q. Le, “Do Better ImageNet Models Transfer Better?”, CVPR 2019. https://arxiv.org/abs/1805.08974 — a 12-dataset study of when source accuracy predicts transfer performance, and where the correlation frays.
  • M. Raghu, C. Zhang, J. Kleinberg, S. Bengio, “Transfusion: Understanding Transfer Learning for Medical Imaging,” NeurIPS 2019. https://arxiv.org/abs/1902.07208 — the when-transfer-fails study: little benefit from ImageNet weights on the medical tasks examined.
  • B. Neyshabur, H. Sedghi, C. Zhang, “What is being transferred in transfer learning?”, NeurIPS 2020. https://arxiv.org/abs/2008.11687 — dissects transfer’s benefit into feature reuse versus low-level statistics and optimization effects.
  • H. Zhang, M. Cisse, Y. Dauphin, D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” ICLR 2018. https://arxiv.org/abs/1710.09412 — the label-blending augmentation; short and easy to implement from the paper.
  • E. Cubuk, B. Zoph, J. Shlens, Q. Le, “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space,” NeurIPS 2020. https://arxiv.org/abs/1909.13719 — the two-knob augmentation policy that made learned augmentation cheap enough for routine use.
  • Dive into Deep Learning: Fine-Tuning (https://d2l.ai/chapter_computer-vision/fine-tuning.html) and Image Augmentation (https://d2l.ai/chapter_computer-vision/image-augmentation.html) — runnable full-scale versions of both halves of this section.
  • torchvision transforms documentation. https://docs.pytorch.org/vision/stable/transforms.html — the standard augmentation toolkit for a project on real images, including the RandAugment implementation.

Before class you should be able to

  • State the four-step transfer recipe and choose between freezing (linear probe) and fine-tuning based on data size and domain similarity.
  • Explain why fine-tuning uses a learning rate 10–100x smaller than training from scratch, and what discriminative learning rates are.
  • Read frozen-trunk code (requires_grad = False, a fresh nn.Linear head, an optimizer over head parameters) and predict what trains.
  • Explain why requires_grad = False does not freeze BatchNorm running statistics, and the .eval() fix.
  • Decide whether a given augmentation is label-preserving or label-destroying for a given task, with examples of each.
  • Distinguish stochastic training augmentation from deterministic preprocessing that validation and test data must retain.
  • Describe what domain shift is and what a negative-transfer result looks like.

Connects to

  • This session is covered by Quiz 2 (Module 2: S09–S12), given at S13.
  • The project guide is released today; this section’s freeze/fine-tune and augmentation decisions are the small-data toolkit most projects lean on.
  • HW2 (out at S12, due at S18) has you rebuild this section’s transfer experiment yourself: convolution and pooling in NumPy, then SmallCNN, a frozen trunk, and head-only training on 24 task-B images.

← S10 · From LeNet to ResNet · All readings · S12 · Training in Practice →