Session S11
Two ways to put knowledge into a model when labels are scarce: through its weights, and through its data.
Deep Learning · CS 4715/5715 · University of Idaho
Graded on experimental design and analysis, not leaderboard position. Start thinking about topics now.
VGG and ResNet (S10) were trained on ImageNet: over a million labeled images.
Knowledge enters through the weights: reuse features learned on a large task as the starting point for a small one.
Knowledge enters through the data: label-preserving transforms encode invariances you know the task has.
Both address the same few-label regime, one through the weights and one through the data; in practice they are combined.
A trained CNN builds its features layer by layer (S09).
The trunk of a network trained on a big dataset is a learned feature extractor; its head belongs to the old task.
A pretrained classifier is a trunk that computes features and a head that maps them to the source task's classes.
nn.Linear(D, K)The recipe has not changed since about 2014. What varies is how much of the trunk you let move.
Build VGG-16 (S10) from the paper's configuration table; import torch and import torch.nn as nn first.
cfg = [64, 64, "M", 128, 128, "M", 256, 256, 256, "M",
512, 512, 512, "M", 512, 512, 512, "M"] # output channels; "M": 2x2 max pool
layers, c_in = [], 3
for v in cfg:
if v == "M":
layers.append(nn.MaxPool2d(2))
else:
layers += [nn.Conv2d(c_in, v, 3, padding=1), nn.ReLU()]
c_in = v
features = nn.Sequential(*layers)
classifier = nn.Sequential(nn.Flatten(), nn.Linear(512 * 7 * 7, 4096),
nn.ReLU(), nn.Dropout(), nn.Linear(4096, 4096), nn.ReLU(),
nn.Dropout(), nn.Linear(4096, 1000))
count = lambda m: sum(p.numel() for p in m.parameters()) # numel: entries per tensor
print(count(features), count(classifier), count(classifier[1]))
14714688 123642856 102764544Trunk, 13 conv layers: 14.7M. Head, 3 fully connected: 123.6M, the first alone 102.8M. Total 138,357,544: the paper's 138M.
Run features (previous page) on two \(224\times224\) images; print the shape and, with count, the parameter count at each max pool.
h, n = torch.randn(2, 3, 224, 224), 0
for layer in features:
h = layer(h)
n += count(layer)
if isinstance(layer, nn.MaxPool2d): # the end of a stage
print(tuple(h.shape), n)
n = 0
(2, 64, 112, 112) 38720
(2, 128, 56, 56) 221440
(2, 256, 28, 28) 1475328
(2, 512, 14, 14) 5899776
(2, 512, 7, 7) 7079424
padding=1 keeps \(H,W\) (S09); each nn.MaxPool2d(2) halves them: \(224/2^5=7\).nn.Linear(25088, 4096).The 102.8M layer exists because the head flattens a \(512\times7\times7\) map into one long vector.
\(123{,}642{,}856/138{,}357{,}544=89.4\%\) of VGG-16 is its classifier. Global average pooling (GAP) averages each channel's \(7\times7\) map to one number.
x = torch.randn(2, 3, 224, 224) # two RGB images
feat = features(x) # features: the VGG-16 page
print(feat.shape)
print(nn.Flatten()(feat).shape) # what the classifier receives
print(feat.mean(dim=(2, 3)).shape) # global average pooling
torch.Size([2, 512, 7, 7])
torch.Size([2, 25088])
torch.Size([2, 512])
feat.mean(dim=(2, 3)) is a reduction over the two spatial axes \(H\) and \(W\): one mean per image and channel. nn.AdaptiveAvgPool2d(1) is the module form of this reduction.
| into the head | the whole head | parameters | |
|---|---|---|---|
| Flatten | \((N,512,7,7)\to(N,25088)\) | VGG-16's three nn.Linear layers | 123,642,856 |
| GAP | mean over \(H,W\): \((N,512)\) | one nn.Linear(512, 1000) | 513,000 |
GoogLeNet and ResNet classify from the averaged vector, so ResNet-152 (about 60M parameters) is smaller than VGG-16 (138M).
The flatten head is sized for \(224\times224\). Feed \(256\times256\) through features and classifier (VGG-16 page).
x2 = torch.randn(2, 3, 256, 256) # a larger image
feat2 = features(x2)
print(feat2.shape) # 256 / 2**5 = 8
gap = nn.AdaptiveAvgPool2d(1) # average to 1x1, whatever H, W arrive
print(gap(feat2).flatten(1).shape) # flatten(1): keep axis 0
print(torch.allclose(gap(feat2).flatten(1), feat2.mean(dim=(2, 3))))
try:
classifier(feat2) # the flatten head
except RuntimeError as e:
print(e)
torch.Size([2, 512, 8, 8])
torch.Size([2, 512])
True
mat1 and mat2 shapes cannot be multiplied (2x32768 and 25088x4096)
Flattening \(8\times8\) gives \(512\cdot64=32{,}768\) inputs where the head expects 25,088. GAP gives 512 numbers whatever the map's size, so a GAP backbone runs at your image size (at least \(32\times32\) for five pools); for transfer, still feed the size it was pretrained at (S12).
Toward ResNet-18, the backbone we transfer: its stages 2–4 halve the resolution and double the channels at their first block. Try the identity skip there (torch, nn as before).
x = torch.randn(8, 64, 56, 56)
branch = nn.Sequential( # F: stride 2, 64 -> 128 channels
nn.Conv2d(64, 128, 3, stride=2, padding=1), nn.ReLU(),
nn.Conv2d(128, 128, 3, padding=1))
print(branch(x).shape, x.shape)
try:
branch(x) + x # the identity skip
except RuntimeError as e:
print(e)
torch.Size([8, 128, 28, 28]) torch.Size([8, 64, 56, 56])
The size of tensor a (28) must match the size of tensor b (56) at non-singleton dimension 3
The identity skip needs equal shapes, so at a downsampling block the skip must change \(X\).
Put a convolution on the skip that produces 128 channels at \(28\times28\) (x and branch from the previous page, count from the VGG-16 page).
shortcut = nn.Conv2d(64, 128, kernel_size=1, stride=2)
print(shortcut(x).shape)
out = branch(x) + shortcut(x) # the projection shortcut
print(out.shape, count(shortcut))
torch.Size([8, 128, 28, 28])
torch.Size([8, 128, 28, 28]) 8320
ResNet projects only where a block changes the shape; every other block keeps the identity skip.
ResNet uses batch normalization (BN) after every convolution. S07's BatchNorm1d on \((N,C)\) normalizes each column over \(N\) rows; BatchNorm2d normalizes each channel over \(N\cdot H\cdot W\) values.
torch.manual_seed(0)
h = torch.randn(8, 16, 5, 5) * 3 + 1 # (N, C, H, W)
bn = nn.BatchNorm2d(16)
mean = h.mean(dim=(0, 2, 3), keepdim=True) # (1, 16, 1, 1)
var = h.var(dim=(0, 2, 3), unbiased=False, keepdim=True)
manual = (h - mean) / torch.sqrt(var + bn.eps)
print(mean.shape, torch.allclose(bn(h), manual, atol=1e-5))
print(count(bn))
torch.Size([1, 16, 1, 1]) True
32
dim=(0, 2, 3): reduce over batch, height, width. keepdim=True keeps \((1,16,1,1)\), which broadcasts back over \((8,16,5,5)\).unbiased=False: divide by the count, \(8\cdot5\cdot5=200\), as BatchNorm does in training mode.Per channel: one mean, one variance, one \(\gamma\), one \(\beta\), shared by all its positions.
Reading 2.2's Block, plus a BatchNorm after each convolution and a projection when the shape changes.
class BasicBlock(nn.Module):
def __init__(self, c_in, c_out, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(c_in, c_out, 3, stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(c_out)
self.conv2 = nn.Conv2d(c_out, c_out, 3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(c_out)
self.shortcut = nn.Identity() # shapes match: no parameters
if stride != 1 or c_in != c_out: # the projection shortcut
self.shortcut = nn.Sequential(
nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out))
def forward(self, x):
out = torch.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
return torch.relu(out + self.shortcut(x))
bias=False: the BatchNorm that follows subtracts each channel's mean, which would cancel a bias; its \(\beta\) supplies the shift.Stack BasicBlock in four stages of two; stages 2–4 halve the resolution and double the channels (count: VGG-16 page).
def resnet(blocks_per_stage, num_classes=1000):
layers = [nn.Conv2d(3, 64, 7, stride=2, padding=3, bias=False), # stem
nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(3, stride=2, padding=1)]
c_in = 64
for i, (c, n) in enumerate(zip([64, 128, 256, 512], blocks_per_stage)):
for j in range(n):
stride = 2 if i > 0 and j == 0 else 1 # first block of stages 2-4
layers.append(BasicBlock(c_in, c, stride))
c_in = c
layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(512, num_classes)]
return nn.Sequential(*layers)
print(count(resnet([2, 2, 2, 2])), count(resnet([3, 4, 6, 3]))) # ResNet-18, -34
11689512 2179767218 weight layers: stem, 16 block convs, final linear; projections not counted.
The transfer recipe on the network just built: everything before the last nn.Linear is the trunk.
net = resnet([2, 2, 2, 2]) # ResNet-18, previous page
trunk, old_head = net[:-1], net[-1] # the head is the last nn.Linear
x = torch.randn(4, 3, 224, 224)
feats = trunk(x)
print(feats.shape, count(trunk), count(old_head))
head = nn.Linear(512, 3) # a fresh head for 3 classes
print(head(feats).shape, count(head))
torch.Size([4, 512]) 11176512 513000
torch.Size([4, 3]) 1539
net[:-1] slices an nn.Sequential: every module but the last. The trunk ends with global average pooling, nn.AdaptiveAvgPool2d(1) (each channel averaged to \(1\times1\)), and nn.Flatten(), so its features are \((N,512)\).The old head had \(512\cdot1000+1000=513{,}000\) parameters. Freeze the trunk and only the new head's \(512\cdot3+3=1{,}539\) train: the linear probe.
A residual block maps \((N,64,32,32)\) to \((N,128,16,16)\). Why does out + x fail, and which projection shortcut fixes it? How many parameters does it add?
The channels (128 against 64) and the resolution (16 against 32) differ. nn.Conv2d(64, 128, kernel_size=1, stride=2) on the skip: \(64\cdot128+128=8{,}320\) parameters.
ResNet's own block (bias=False, then BatchNorm2d(128)) adds \(64\cdot128+2\cdot128=8{,}448\).
A RuntimeError at out + x means the block changed \(H,W\) or \(C\): put a \(1\times1\) convolution with the block's stride on the skip.
Once depth stopped being the obstacle, depth, width and resolution became separate choices (reading 2.2).
| network | change | reported result |
|---|---|---|
| Pre-activation ResNet (He et al., ECCV 2016) | BN and ReLU before each conv; nothing after the addition | 1001 layers trained on CIFAR-10, with improved accuracy |
| Wide ResNet (Zagoruyko, Komodakis, BMVC 2016) | fewer layers, more channels | on CIFAR, 16 wide layers outperform thousand-layer thin nets |
| DenseNet (Huang et al., CVPR 2017) | each layer gets all earlier outputs, concatenated | ResNet-level ImageNet accuracy at about half the parameters and compute |
| EfficientNet (Tan, Le, ICML 2019) | depth, width and resolution scaled jointly | B0: 76.3% ImageNet top-1 with 5.3M parameters |
| ConvNeXt (Liu et al., CVPR 2022) | ResNet-50 modernized step by step: recipe, \(7\times7\) kernels, LayerNorm | matches or exceeds comparable Swin Transformers |
Pre-activation's design rule: anything placed on the identity path, such as a ReLU or a BatchNorm, multiplies every gradient that crosses it.
ResNet-18's trunk has 11.2M parameters and the new head 1,539. How many of them should train?
| what trains | when | risk | |
|---|---|---|---|
| Freeze (linear probe) | the head only | few labels; target resembles the source | features may not suit an unusual domain |
| Fine-tune | head and some or all of the trunk, at a learning rate \(10\text{–}100\times\) smaller than from scratch | more labels, or a different domain (medical scans, spectrograms) | a large learning rate destroys pretrained features |
| Middle path | the last block or two, plus the head | in between |
Default: freeze first, and unfreeze once the data can support it.
No downloads: reading 2.3 builds the pretrained network itself, on \(16\times16\) grayscale images with Gaussian noise (standard deviation 0.4) added.
class 0
class 1
class 2
class 3
class 0: "+"
class 1: "x"
Noise-free \(9\times9\) crops drawn with the reading's draw_bar; each bar is 7 pixels long at a random position.
A "+" is bars of classes 0 and 2; an "x" is bars of classes 1 and 3. A trunk that detects task A's four orientations already computes the features task B needs.
One architecture for the pretrained network, the transferred model and the from-scratch baseline.
import torch
import torch.nn as nn
def make_trunk():
return nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), # (N, 8, 8, 8)
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), # (N, 16, 4, 4)
nn.Flatten(), # (N, 256)
)
count = lambda m: sum(p.numel() for p in m.parameters())
trunk, head = make_trunk(), nn.Linear(256, 2)
x = torch.randn(24, 1, 16, 16) # 24 grayscale 16x16 images
feats = trunk(x)
print(feats.shape, head(feats).shape)
print(count(trunk), count(head), count(trunk) + count(head))
torch.Size([24, 256]) torch.Size([24, 2])
1248 514 1762
Trunk: \(80+1{,}168=1{,}248\) parameters. Head: \(256\cdot2+2=514\). HW2 (out at S12) calls this architecture SmallCNN.
For each seed, reading 2.3 seeds torch with it and builds these tensors with its make_task_a and make_task_b; one short function trains every model.
| tensors | shapes | contents |
|---|---|---|
Xa, ya | (2000, 1, 16, 16), (2000,) | task A images; labels 0–3 |
Xb, yb | (24, 1, 16, 16), (24,) | task B training images; labels 0 ("+"), 1 ("x") |
Xb_te, yb_te | (500, 1, 16, 16), (500,) | task B test images |
def train(model, X, y, epochs, params=None):
params = list(model.parameters()) if params is None else list(params)
opt = torch.optim.Adam(params, lr=1e-3) # updates only these params
loss_fn = nn.CrossEntropyLoss()
for _ in range(epochs):
perm = torch.randperm(len(X)) # new order every epoch
for i in range(0, len(X), 64): # minibatches of 64
idx = perm[i:i + 64]
opt.zero_grad()
loss_fn(model(X[idx]), y[idx]).backward()
opt.step()
Inside reading 2.3's loop over seeds 0–2, which builds Xa, ya, Xb, yb with make_task_a and make_task_b; make_trunk and train come from the previous pages.
pre = nn.Sequential(make_trunk(), nn.Linear(256, 4))
train(pre, Xa, ya, epochs=8) # 1) pretrain trunk + 4-way head on task A
trunk = pre[0] # 2) transfer: keep the trunk ...
for p in trunk.parameters():
p.requires_grad = False # ... freeze it ...
torch.manual_seed(seed + 100)
head = nn.Linear(256, 2) # ... and train a fresh 2-way head
transfer = nn.Sequential(trunk, head)
train(transfer, Xb, yb, epochs=150, params=head.parameters())
torch.manual_seed(seed + 100) # 3) baseline: same architecture, from scratch
scratch = nn.Sequential(make_trunk(), nn.Linear(256, 2))
train(scratch, Xb, yb, epochs=150)
requires_grad = False: backward computes no gradient for the trunk, and the optimizer, built over head.parameters(), updates only the head's 514 parameters.Test accuracy on the 500 task-B test images (reading 2.3's printout):
seed 0: transfer = 0.7640, scratch = 0.5940
seed 1: transfer = 0.7740, scratch = 0.5380
seed 2: transfer = 0.9480, scratch = 0.8800
transfer mean test accuracy = 0.8287
scratch mean test accuracy = 0.6707
Seed 0's trunk: the first convolution's 8 kernels of \(3\times3\), before and after 8 epochs on task A (one kernel per column, one color scale; white is 0). Orange: positive weights; blue: negative.
The frozen trunk builds its 256 features from these detectors; the head's 514 parameters only learn to read them.
Reading 2.3 inverts the contrast of task B (X = -X: dark shapes on bright noise). Shapes, labels and difficulty stay the same.
matched
shifted
matched domain frozen transfer = 0.8287, scratch = 0.6707
shifted domain (inverted) frozen transfer = 0.6347, scratch = 0.6627
Transfer helps most when your images resemble the source task's and labels are scarce. In an unusual domain, measure the scratch baseline.
Training on \((a(x),y)\) asserts that \(a\) does not change the class.
Each family asserts a different invariance; pick only those that hold for your labels.
| Geometric | shifts, crops (random-resized crops are the ImageNet staple), horizontal flips, small rotations, scaling |
| Photometric | brightness, contrast, saturation and hue jitter; grayscale; blur; noise |
| Occlusion, mixing | random erasing (cutout); mixup blends two images and their labels; CutMix pastes a labeled patch |
| Learned policies | AutoAugment, RandAugment: tune the type and strength |
Strength matters as much as type: a transform that changes the class is label noise.
Reading 2.3: class 0 is an "L", class 1 its mirror image, at random positions in \(16\times16\) noise; 32 training examples.
class 0: "L"
class 1: mirrored "L"
class 0, shifted:
still an "L", label 0 correct
class 0, flipped:
a mirrored "L", label 0 wrong
Noise-free rows 2–13 of the \(16\times16\) images, drawn with the reading's draw_L, torch.roll and torch.flip.
A shift keeps the class. A horizontal flip turns one class into the other and keeps the old label.
X has shape \((32,1,16,16)\), that is \((N,C,H,W)\); rng is a NumPy random generator.
def shift_aug(X, rng):
"""Random translation up to 3 px: label-PRESERVING for this task."""
out = torch.empty_like(X)
for i in range(len(X)):
dr, dc = rng.integers(-3, 4, size=2)
out[i] = torch.roll(X[i], shifts=(int(dr), int(dc)), dims=(1, 2))
return out
def flip_aug(X, rng):
"""Random horizontal flip: label-DESTROYING for this task."""
out = X.clone()
mask = torch.from_numpy(rng.random(len(X)) < 0.5)
out[mask] = torch.flip(out[mask], dims=[3])
return out
X[i] is one image, \((1,16,16)\); dims=(1, 2) rolls its rows and columns, wrapping around the edge.mask: booleans, shape (32,). out[mask] is about half the images, \((k,1,16,16)\); dims=[3] reverses their width axis.Reading 2.3 trains a CNN with the layers of make_trunk plus a 2-way head, 400 Adam steps on the 32 images, with a fresh augmentation at every step, and tests on 500 images; 3 seeds.
no augmentation test acc = 0.7147 (seeds: 0.706, 0.634, 0.804)
random shifts (label-preserving) test acc = 0.9633 (seeds: 1.000, 0.896, 0.994)
horizontal flips (label-destroying) test acc = 0.5173 (seeds: 0.510, 0.462, 0.580)
Same code, same effort. The difference is whether the invariance you asserted holds for this task.
Real datasets carry chirality (a shape and its mirror image are different classes) and orientation semantics more often than expected.
An augmentation is valid only if it preserves the label for your task.
You have 40 labeled images per class of 3 road signs, including "turn left" and "turn right", and an ImageNet-pretrained network whose trunk outputs features of shape \((N,512)\).
(a) Freeze or fine-tune first? How many parameters does the new head train?
(b) Which of these augmentations is unsafe here: small rotations, brightness jitter, horizontal flips? Why?
(a) Freeze first: with 40 images per class, a linear probe is the default. nn.Linear(512, 3) trains \(512\cdot3+3=1{,}539\) parameters.
(b) Horizontal flips: they turn a "turn left" sign into a "turn right" sign and keep the old label. Small rotations and brightness jitter keep the class.