S10 Reading — From LeNet to ResNet | Deep Learning

Module 2 · Reading 2.2 · read before S10: CNNs II — ResNet & Modern Architecture Design

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

Section 2.1 gave you the vocabulary of convolutional networks: kernels, stride, padding, pooling, channels. An architecture is a way of spending that vocabulary — how many layers, how wide, where to downsample. Between 1998 and 2015 the field went from 5-layer networks reading digits to 152-layer networks surpassing a reported human top-5 error estimate on ImageNet classification (an informal single-annotator baseline, not a claim that machines see better than people), and the story is not one of steady scaling. For most of that period, adding layers made networks harder to train, and each architectural generation exists because it removed specific obstacles.

This section walks that history (LeNet-5, AlexNet, VGG, ResNet) but treats it as an engineering argument rather than a timeline. The central question is: why was depth hard, and what fixed it? You will reproduce the core failure (gradients vanishing through deep stacks) with real numbers, then build the fix (the residual connection) and measure exactly what it changes.

How to read this

LeNet-5: the blueprint (1998)
AlexNet and the 2012 moment
The architecture zoo: depth is not parameters
Why depth was hard: the vanishing gradient, measured
Batch normalization: what it actually buys
The degradation experiment, reproduced in miniature
ResNet: learn the residual
After ResNet: scaling as a discipline
Going deeper
What to take from the history
Common pitfalls
Try it yourself
Further reading

LeNet-5: the blueprint (1998)

LeNet-5, published by LeCun, Bottou, Bengio, and Haffner in 1998 after a decade of refinement at Bell Labs, already contains the modern template: two rounds of convolution and downsampling to build increasingly abstract feature maps, followed by fully connected layers to classify them. It read 32x32 grayscale digits, used 5x5 kernels and average-pooling-like subsampling, tanh-family activations, and about 60,000 parameters. The 1998 paper describes the system’s commercial deployment for reading checks: this was production technology, not a lab demo.

What LeNet did not have was headroom. Scaling the recipe up hit three obstacles at once: compute (CPUs of the era could not train much bigger networks in reasonable time), data (datasets like MNIST’s 60,000 training images cannot support many more parameters without overfitting), and trainability (deeper stacks trained by plain backpropagation through saturating activations fail to train, for reasons quantified below). For roughly a decade, convolutional networks remained a specialty while the broader field favored methods like SVMs with engineered features, which, on the benchmarks of that time, often won.

AlexNet and the 2012 moment

Two of the obstacles fell together. ImageNet (Deng et al., 2009), through its yearly challenge (ILSVRC, from 2010), supplied about 1.2 million labeled training images across 1,000 classes; commodity GPUs supplied several times the peak arithmetic of a contemporary CPU, ran the large matrix products of training far faster still, and had several orders of magnitude more throughput than the workstations LeNet was trained on. AlexNet (Krizhevsky, Sutskever, and Hinton, published at NeurIPS 2012) was architecturally a scaled LeNet: five convolutional layers plus three fully connected, about 60 million parameters, trained for about a week on two GTX 580 GPUs. Its non-obvious choices became standard practice: ReLU activations instead of saturating tanh/sigmoid (directly attacking the gradient problem below), dropout in the classifier layers, and aggressive data augmentation (crops, flips, color perturbation — the subject of Section 2.3).

Historical note. In the ILSVRC-2012 competition, AlexNet’s top-5 error of 15.3% (the number reported in the paper) stood against 26.2% for the runner-up built on engineered features — an unheard-of margin in a mature benchmark competition. The result’s reception is one of the field’s well-documented turning points: within about two years, essentially every competitive entry was a deep convolutional network, and computer vision reorganized around learned features. When people date the deep learning era, this is the event they usually point to.

VGG (Simonyan and Zisserman, ICLR 2015) then showed how far discipline alone could go: use only 3x3 kernels, double the channel count after each 2x2 max pool up to 512, and push to 16–19 weight layers. As computed in Section 2.1, stacked 3x3 kernels match larger kernels’ receptive fields with fewer parameters and more nonlinearities. But VGG also marked a ceiling: its authors, and everyone else who tried, found that stacking still more plain layers stopped helping. To see why, you need to look at what depth does to gradients.

The architecture zoo: depth is not parameters

Before the gradient story, one structural misconception is worth killing with numbers: deeper does not mean more parameters, and the interesting design work is in where the parameters sit. The table collects the headline members of the zoo, using the papers’ own reported figures where available (the “~” entries are widely reported counts rather than numbers printed in the original paper):

Network	Venue, year	Weight layers	Parameters	Signature idea
LeNet-5	Proc. IEEE 1998	5	~60 K	conv + subsample template
AlexNet	NeurIPS 2012	8	60 M (paper)	ReLU, dropout, GPU training
VGG-16	ICLR 2015	16	138 M (paper)	uniform 3x3 stacks
GoogLeNet	CVPR 2015	22	~5 M	multi-scale Inception modules, no big FC head
ResNet-152	CVPR 2016	152	~60 M	residual connections
DenseNet-121	CVPR 2017	121	~8 M (paper)	every layer sees all earlier features
EfficientNet-B0	ICML 2019	—	5.3 M (paper)	jointly scaled width/depth/resolution

(Venue abbreviations: NeurIPS — Neural Information Processing Systems; ICLR — International Conference on Learning Representations; CVPR — Conference on Computer Vision and Pattern Recognition; ICML — International Conference on Machine Learning.)

Read down the parameter column against the depth column. VGG-16, at 16 layers, is more than twice the size of 152-layer ResNet-152. GoogLeNet (the Inception network from Szegedy and colleagues at Google, its name a nod to LeNet) reached 22 layers with on the order of five million parameters, more than twenty times fewer than VGG-16 published the same year. Depth and size decoupled early, and the papers’ stated motivations are explicitly about efficiency: Inception’s multi-scale modules were designed to stay within a fixed compute budget, and DenseNet’s dense connectivity exists to let layers reuse earlier features instead of relearning them.

Where does VGG’s bulk actually live? Build it and look. The script constructs VGG-16 exactly from the paper’s configuration table (thirteen 3x3 conv layers in five pooled stages, then three fully connected layers) and counts:

import torch
import torch.nn as nn

torch.manual_seed(0)

# VGG-16, built from the paper's configuration table (Simonyan & Zisserman,
# ICLR 2015, configuration D): numbers are output channels, 'M' is 2x2 max pool.
cfg = [64, 64, "M", 128, 128, "M", 256, 256, 256, "M",
       512, 512, 512, "M", 512, 512, 512, "M"]

features, c_in = [], 3
for v in cfg:
    if v == "M":
        features.append(nn.MaxPool2d(2))
    else:
        features += [nn.Conv2d(c_in, v, 3, padding=1), nn.ReLU()]
        c_in = v
features = nn.Sequential(*features)

classifier = nn.Sequential(               # input image 224x224 -> 512 x 7 x 7
    nn.Flatten(),
    nn.Linear(512 * 7 * 7, 4096), nn.ReLU(), nn.Dropout(),
    nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(),
    nn.Linear(4096, 1000),
)

def n_params(m):
    return sum(p.numel() for p in m.parameters())

conv_p, fc_p = n_params(features), n_params(classifier)
print(f"conv layers (13):       {conv_p:12,d} params")
print(f"fully connected (3):    {fc_p:12,d} params")
print(f"total:                  {conv_p + fc_p:12,d} params")
print(f"share held by the FC head: {fc_p / (conv_p + fc_p):.1%}")
print(f"first FC layer alone:   {512 * 7 * 7 * 4096 + 4096:12,d} params")

Running this prints:

conv layers (13):         14,714,688 params
fully connected (3):     123,642,856 params
total:                   138,357,544 params
share held by the FC head: 89.4%
first FC layer alone:    102,764,544 params

The total, 138,357,544, reproduces the 138M the VGG paper reports, always a good sign when a from-scratch reconstruction hits the published number. And the breakdown explains the zoo table’s oddity: 89.4% of VGG-16 is its classifier head, and a single layer (flattened 7x7x512 features into 4096 units) holds 102.8 million parameters, seven times all thirteen conv layers combined. This is the Section 2.1 dense-versus-conv arithmetic striking inside a real architecture. GoogLeNet and ResNet escaped it with global average pooling: average each final feature map to one number and classify from that vector, deleting the giant flatten-and-project. That one decision, more than anything about depth, is why 152 layers can cost less than 16.

Historical note. The zoo grew up inside a competition. The ImageNet Large Scale Visual Recognition Challenge ran annually from 2010 to 2017, and its leaderboard is a time-lapse of this section: engineered features (2010–2011), AlexNet (2012), Clarifai and VGG/GoogLeNet (2013–2014), ResNet (2015). After 2017 the organizers wound the challenge down — top-5 error had fallen below three percent, and the benchmark had done its work. The architectures outlived the contest: the zoo’s members became the standard pretrained backbones of Section 2.3.

Why depth was hard: the vanishing gradient, measured

Backpropagation multiplies. The gradient at layer 1 of a deep stack is a product with one factor per layer above it, each factor involving that layer’s weights and the derivative of its activation. If those factors are typically smaller than 1, the product shrinks geometrically with depth. Sigmoid activations make this nearly unavoidable: the sigmoid’s derivative is at most 0.25, so each layer contributes a factor of at most a quarter (times the weights). Hochreiter’s 1991 thesis and Bengio, Simard, and Frasconi’s 1994 paper analyzed exactly this failure, in recurrent networks; deep feedforward stacks inherit it wholesale.

Here it is happening, in five network depths:

import torch
import torch.nn as nn

torch.manual_seed(0)

def make_plain_net(depth, width=32):
    """A deep stack of Linear + Sigmoid layers (early-1990s style)."""
    layers = []
    for _ in range(depth):
        layers += [nn.Linear(width, width), nn.Sigmoid()]
    layers += [nn.Linear(width, 1)]
    return nn.Sequential(*layers)

x = torch.randn(64, 32)
y = torch.randn(64, 1)

for depth in [2, 6, 12, 24, 48]:
    torch.manual_seed(0)
    net = make_plain_net(depth)
    loss = nn.functional.mse_loss(net(x), y)
    loss.backward()
    first = net[0]   # the layer closest to the input
    last = net[-1]   # the output layer
    g_first = first.weight.grad.norm().item()
    g_last = last.weight.grad.norm().item()
    print(f"depth {depth:2d}: ||grad|| at output layer = {g_last:.2e}, "
          f"at first layer = {g_first:.2e}, ratio = {g_first / g_last:.1e}")

Running this prints:

depth  2: ||grad|| at output layer = 4.18e-01, at first layer = 1.68e-02, ratio = 4.0e-02
depth  6: ||grad|| at output layer = 1.48e+00, at first layer = 4.70e-06, ratio = 3.2e-06
depth 12: ||grad|| at output layer = 8.26e-01, at first layer = 2.05e-11, ratio = 2.5e-11
depth 24: ||grad|| at output layer = 7.76e-02, at first layer = 5.30e-21, ratio = 6.8e-20
depth 48: ||grad|| at output layer = 1.97e+00, at first layer = 0.00e+00, ratio = 0.0e+00

Read the last two rows carefully. At depth 24 the first-layer gradient norm is 5.30e-21; at depth 48 it prints as 0.00e+00. The gradient itself is not zero: its entries are around 2e-42, so small that float32 can hold them only as subnormal numbers, and .norm() squares them before summing. The squares fall below the smallest positive float32 (about 1.4e-45), so the norm underflows to exactly zero; in float64, net[0].weight.grad.double().norm() gives about 1.4e-41. Either way the layer does not learn: an update lr * grad of that size is far below the spacing between neighbouring float32 numbers near the weights (about 7e-9 near 0.1), so W - lr * grad equals W exactly at any practical learning rate. Meanwhile the output layer’s gradient stays healthy at every depth. The network is not “training slowly” at the bottom; below a certain depth it is not training at all. Note the ratio column: each factor-of-2 increase in depth squares the attenuation, exactly the geometric decay the multiplication argument predicts.

The 2010–2015 period produced partial fixes, several covered in Section 1.7: ReLU activations (derivative exactly 1 where active), variance-preserving initialization (Glorot 2010, He 2015), and batch normalization (Ioffe and Szegedy, 2015), which re-standardizes activations so each layer sees inputs at a trainable scale. With them, plain networks of VGG depth train from scratch (VGG itself, from 2014, predates He initialization and batch normalization). They were not enough beyond that depth.

Batch normalization: what it actually buys

Batch normalization deserves a closer look before ResNet, because ResNet is built on top of it: the original ResNet places a BN layer after every convolution, and the paper is explicit that its plain and residual networks alike train with BN throughout. The mechanism: for each channel, standardize the activations across the current minibatch (subtract the batch mean, divide by the batch standard deviation), then apply a learned scale \(\gamma\) and shift \(\beta\). The layer’s output distribution is therefore pinned at a trainable operating point regardless of what the weights upstream are doing.

What that buys is scale invariance to bad initialization, among other things, and you can measure it. The script builds a 30-block ReLU stack whose weights are deliberately under-scaled (multiplied by 0.5 after standard initialization, the same trick as the residual experiment below), and probes the standard deviation of activations at blocks 1, 10, 20, and 30, plus the gradient reaching the first layer:

import torch
import torch.nn as nn

torch.manual_seed(0)

def make_stack(depth, use_bn, scale, width=64):
    layers = []
    for _ in range(depth):
        layers.append(nn.Linear(width, width))
        if use_bn:
            layers.append(nn.BatchNorm1d(width))
        layers.append(nn.ReLU())
    net = nn.Sequential(*layers)
    with torch.no_grad():
        for p in net.parameters():
            if p.dim() == 2:            # weight matrices only
                p.mul_(scale)           # deliberately under-scaled init
    return net

def probe(depth, use_bn, scale=0.5):
    torch.manual_seed(0)
    net = make_stack(depth, use_bn, scale)
    x = torch.randn(256, 64)
    # Record activation std after selected layers.
    h, stds = x, {}
    per_block = 3 if use_bn else 2
    for i, layer in enumerate(net):
        h = layer(h)
        block = i // per_block + 1
        if isinstance(layer, nn.ReLU) and block in (1, 10, 20, 30):
            stds[block] = h.std().item()
    loss = net(x).pow(2).mean()
    loss.backward()
    g1 = net[0].weight.grad.norm().item()
    tag = "with BN   " if use_bn else "without BN"
    print(f"{tag}: act std at block 1/10/20/30 = "
          + "/".join(f"{stds[b]:.1e}" for b in (1, 10, 20, 30))
          + f", first-layer grad norm = {g1:.1e}")

probe(30, use_bn=False)
probe(30, use_bn=True)

Running this prints:

without BN: act std at block 1/10/20/30 = 1.7e-01/4.5e-02/4.1e-02/4.1e-02, first-layer grad norm = 0.0e+00
with BN   : act std at block 1/10/20/30 = 5.8e-01/6.1e-01/6.1e-01/6.0e-01, first-layer grad norm = 3.2e+01

Without BN, the activation scale collapses within ten blocks, from 0.17 to 0.045, and then flatlines around 0.04, which is the biases talking: by that depth the input’s contribution has shrunk to nearly nothing and each layer mostly outputs a function of its own bias. The backward pass through those under-scaled weights attenuates just as the sigmoid stack’s did: the first-layer gradient’s entries are at most about 1e-23, and its float32 norm underflows to exact zero (float64 gives about 1.3e-22). With BN, every block’s activations sit near 0.6 (BN standardizes the pre-activations to unit scale; the ReLU then trims that to about 0.58) no matter the depth, and the first-layer gradient is a healthy 32. The 0.5 mis-scaling that killed the plain stack is invisible to the normalized one — each BN layer simply divides it back out.

So why was BN not the end of the story? Because keeping signals at scale fixes the conditioning of deep training, not its optimization geometry. The ResNet paper’s evidence is exactly that distinction: plain networks with BN throughout, where nothing vanishes, in which depth still hurts. The miniature below reproduces the symptom at toy scale, with one caveat stated after its results.

Pitfall. The failure of very deep plain networks is not overfitting. The ResNet paper reports that on CIFAR-10 a 56-layer plain network has higher training error than a 20-layer one — the deeper model fits the data worse, despite strictly more capacity, and despite BatchNorm keeping forward signals healthy. He et al. call this the degradation problem. It is a failure of optimization: a 56-layer network trivially contains the 20-layer solution (set 36 layers to compute the identity), but gradient descent through stacked nonlinear layers does not find it. Diagnosing “deeper got worse” as overfitting (and reaching for regularization) treats the wrong disease.

The degradation experiment, reproduced in miniature

The degradation claim is strange enough to deserve a demonstration you can run. The design mirrors He et al.’s CIFAR-10 experiment at toy scale: identical blocks, identical data, identical training budget; the only variables are the number of blocks and whether each block has a skip connection. To make “fitting” unambiguous, the metric is training accuracy on the XOR-quadrant task (label 1 when \(x_1 x_2 \gt 0\)), a task a one-block network handles. If deeper plain networks fit the training data worse than shallow ones, that is degradation; overfitting cannot explain it, because overfitting means fitting the training data better.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_num_threads(1)

# The degradation experiment in miniature: does adding depth make the
# TRAINING fit worse? Plain vs residual blocks, four depths, same budget.

class Block(nn.Module):
    def __init__(self, width, skip):
        super().__init__()
        self.fc1 = nn.Linear(width, width)
        self.fc2 = nn.Linear(width, width)
        self.relu = nn.ReLU()
        self.skip = skip

    def forward(self, x):
        out = self.fc2(self.relu(self.fc1(x)))
        if self.skip:
            out = out + x
        return self.relu(out)

def make_data(n, rng):
    X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
    y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)     # XOR-quadrant task
    X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
    return torch.from_numpy(X), torch.from_numpy(y)

def train_accuracy(n_blocks, skip, seed, width=32, steps=600):
    rng = np.random.default_rng(seed)
    torch.manual_seed(seed)
    X, y = make_data(256, rng)
    net = nn.Sequential(nn.Linear(2, width), nn.ReLU(),
                        *[Block(width, skip) for _ in range(n_blocks)],
                        nn.Linear(width, 2))
    opt = torch.optim.SGD(net.parameters(), lr=0.05, momentum=0.9)
    loss_fn = nn.CrossEntropyLoss()
    for _ in range(steps):
        idx = torch.randint(0, len(X), (64,))
        opt.zero_grad()
        loss_fn(net(X[idx]), y[idx]).backward()
        opt.step()
    with torch.no_grad():
        return (net(X).argmax(1) == y).float().mean().item()

print(f"{'blocks':>6} {'plain train acc':>16} {'residual train acc':>19}")
for n in [1, 4, 16, 32]:
    row = []
    for skip in [False, True]:
        row.append(np.mean([train_accuracy(n, skip, seed) for seed in range(3)]))
    print(f"{n:6d} {row[0]:16.4f} {row[1]:19.4f}")

Running this prints:

blocks  plain train acc  residual train acc
         0.8815              0.8854
         0.5091              0.8841
         0.5221              0.8906
         0.5221              0.8880

At one block, plain and residual are indistinguishable: 0.88 either way, and the task’s label noise means 0.88-0.89 is about as fit as this data gets. Then the plain column falls off a cliff: at 4 blocks, training accuracy drops to 0.51 and stays there at 16 and 32 blocks. The 4-block plain network contains a good shallow solution in principle, yet SGD does not find it under this recipe. The caveat: this recipe keeps PyTorch’s default initialization and has no normalization, so before any training the plain network’s first-layer gradient is already tiny, the attenuation measured earlier in this section. The miniature therefore shows the degradation symptom, a deeper plain network fitting its own training data worse, with vanishing gradients mixed in; the paper’s plain networks had BN throughout. The residual column remains near 0.88–0.89 because these residual branches initialize and optimize near a small perturbation of identity in this experiment. Extra blocks are not inherently harmless; that claim depends on keeping the residual branch initially small and the stack well scaled.

ResNet: learn the residual

The fix, from He, Zhang, Ren, and Sun (arXiv December 2015, CVPR 2016), is one line of arithmetic. Instead of asking a block of layers to learn a mapping \(H(x)\) directly, let the block compute a correction \(F(x)\) and add the input back:

\[\text{out} = F(x) + x\]

The added identity path, the skip connection or residual connection, changes the default. If \(F(x)\) starts near zero, a residual block starts near identity. In the backward pass, \(d(\text{out})/dx = I + dF/dx\), so each block contributes a direct identity term in addition to the residual derivative. Across many blocks the product of these Jacobians can still shrink, explode, or cancel in some directions; residual structure improves the path rather than guaranteeing an undiminished gradient.

Here is a small ResNet-style block, and a controlled comparison. To recreate 2015-era conditions where each block attenuates signal, the conv weights are scaled down after standard initialization; the only difference between the two configurations is the out + x:

import torch
import torch.nn as nn

torch.manual_seed(0)

class Block(nn.Module):
    """Two 3x3 convs. With skip=True this is a basic ResNet block."""
    def __init__(self, channels, skip):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.relu = nn.ReLU()
        self.skip = skip

    def forward(self, x):
        out = self.conv2(self.relu(self.conv1(x)))
        if self.skip:
            out = out + x          # the residual connection
        return self.relu(out)

def grad_at_first_layer(n_blocks, skip, scale):
    torch.manual_seed(0)
    blocks = nn.Sequential(*[Block(8, skip) for _ in range(n_blocks)])
    # Shrink the conv weights so each block is a contraction, as
    # under-scaled initializations were before He init (2015).
    with torch.no_grad():
        for p in blocks.parameters():
            if p.dim() == 4:
                p.mul_(scale)
    x = torch.randn(4, 8, 8, 8)
    out = blocks(x)
    loss = out.pow(2).mean()
    loss.backward()
    return blocks[0].conv1.weight.grad.norm().item()

print(f"{'blocks':>6} {'plain':>12} {'residual':>12}")
for n in [2, 8, 16, 32]:
    g_plain = grad_at_first_layer(n, skip=False, scale=0.5)
    g_res = grad_at_first_layer(n, skip=True, scale=0.5)
    print(f"{n:6d} {g_plain:12.2e} {g_res:12.2e}")

Running this prints:

blocks        plain     residual
   2.44e-05     5.23e-02
   4.39e-14     5.86e-02
   0.00e+00     5.42e-02
   0.00e+00     6.36e-02

Two columns, two completely different regimes. The plain stack’s first-layer gradient collapses from 1e-05 to 1e-14 to float32 zero as blocks are added. The residual stack’s first-layer gradient sits between 5.2e-02 and 6.4e-02 at every depth — 2 blocks or 32, it barely moves. Under this recipe the skip connection did not merely reduce the attenuation, it took depth out of it. Keep the claim scoped to the experiment: as the section above says, the general statement is weaker. Residual structure improves the gradient path, and these branches happen to initialize and stay near a small perturbation of the identity.

The results were immediate: ResNets trained at 152 layers, won the ILSVRC 2015 classification task (the paper reports 3.57% top-5 error for an ensemble), and the residual connection became a fixture of network design far beyond vision. The Transformers of Chapter 3 place one around every attention and MLP sublayer, for exactly the reasons your gradient table shows.

People. Kaiming He was a researcher at Microsoft Research Asia when he and colleagues Xiangyu Zhang, Shaoqing Ren, and Jian Sun developed ResNet; the same group produced the He initialization scheme and, later at Facebook AI Research, work spanning detection (Mask R-CNN) and self-supervised learning (MoCo, MAE). The ResNet paper became one of the most cited papers in all of science within a decade of publication — a reasonable proxy for how thoroughly the residual connection permeated deep learning.

After ResNet: scaling as a discipline

Once depth stopped being the obstacle, the question became how to spend a compute budget well, and the post-2015 architectures are best read as answers to that question.

DenseNet (Huang, Liu, van der Maaten, and Weinberger, CVPR 2017) pushed feature reuse to its limit: within a block, every layer receives the concatenated outputs of all earlier layers, so nothing needs relearning. The paper reports ImageNet accuracy comparable to ResNets at roughly half the parameters and FLOPs — the zoo table’s DenseNet-121 at about 8 million parameters is that claim in one number. Wide ResNets (Zagoruyko and Komodakis, BMVC [British Machine Vision Conference] 2016) pushed the other direction, showing that on CIFAR-scale benchmarks a shallow-but-wide residual network could outperform extremely deep thin ones: the paper’s headline result pits a 16-layer wide network against thousand-layer competitors. Depth, width, and input resolution had become three independent design choices, each with diminishing returns when scaled alone.

EfficientNet (Tan and Le, ICML 2019) made that observation quantitative. Starting from a small base network found by neural architecture search, the authors scale depth, width, and resolution jointly by fixed ratios: a single “compound coefficient” sets all three. The paper reports that the base model, EfficientNet-B0, reaches 76.3% ImageNet top-1 accuracy with 5.3 million parameters, and the scaled-up B7 reaches 84.4% while being 8.4x smaller than the best prior network at that accuracy. The deeper lesson generalizes far beyond vision: balanced scaling beats scaling any single dimension, a theme that returns as the scaling laws of Section 3.5.

The coda, for now, is ConvNeXt (Liu et al., CVPR 2022), a paper written after Vision Transformers (Section 3.8) had displaced CNNs from vision leaderboards. Its authors took a standard ResNet-50 and modernized it step by documented step: ViT-style training recipes (stronger augmentation, AdamW, longer schedules), larger 7x7 depthwise kernels, fewer normalizations and activations, LayerNorm in place of BatchNorm. Each step is ablated in the paper, and the finished network (still a pure convolutional model, built from Section 2.1’s parts list) matches or exceeds comparable Swin Transformers on the benchmarks the paper reports. The instructive point is not “CNNs won after all”; it is how much of the apparent architecture gap was really a training recipe gap — which is exactly the terrain of Section 2.4.

Going deeper

The residual story as told above is the standard account, and it is broadly right, but research since 2016 has refined why skips work, in ways worth knowing before you read current papers.

Where the addition happens matters. He and colleagues’ follow-up, “Identity Mappings in Deep Residual Networks” (ECCV [European Conference on Computer Vision] 2016), analyzed variants of the block and found that keeping the skip path completely clean (no ReLU, no BN, nothing after the addition) trains best; their “pre-activation” ordering (BN and ReLU before each conv) let them train a 1001-layer network on CIFAR-10 with improved accuracy. The design rule that survives: anything you insert on the identity path, such as a ReLU or a BatchNorm, multiplies every gradient that crosses it.

Residual networks behave like ensembles of shallow paths. Expand \((F + I)\) compositions and a 32-block ResNet is a sum over 2^32 paths of different lengths; the number of paths through \(k\) blocks is the binomial coefficient \(\binom{32}{k}\), so most paths pass through about half of the blocks. Veit, Wilber, and Belongie (NeurIPS 2016) showed experimentally that trained ResNets lean on this structure: deleting individual blocks at test time barely hurts (deleting a layer from VGG is catastrophic, in their same experiments), and most gradient arrives via paths only a few blocks long. On this reading, a deep ResNet trains like a collection of shallow networks that share weights — which is why its gradients look depth-independent, as your table showed. Stochastic depth (Huang et al., ECCV 2016), which randomly drops whole blocks during training, both exploits and corroborates the picture.

Skips reshape the loss surface literally. Li and colleagues (NeurIPS 2018) computed 2-D visualizations of loss landscapes and found that deep plain networks show jagged, chaotic surfaces where the residual versions of the same networks look smooth and nearly convex around their minima. Visualization has caveats (two dimensions out of millions), but it matches the optimization behavior you measured.

And BN’s textbook explanation has been revised. Ioffe and Szegedy named their paper after “internal covariate shift”: the drifting of layer input distributions during training. Santurkar, Tsipras, Ilyas, and Madry (NeurIPS 2018) constructed experiments where covariate shift is deliberately reinjected after BN and found training barely affected; their evidence points instead to BN smoothing the optimization landscape (more predictable gradients, tolerance of larger learning rates). The practical guidance is unchanged (use normalization), but the episode is a good calibration for reading deep learning papers: a method can be indispensable while its original explanation is wrong, and the field can take years to notice.

What to take from the history

The through-line is worth stating plainly, because it recurs in every later chapter. LeNet established the pattern; what stood between it and AlexNet was mostly resources (data and compute); what stood between VGG and ResNet was purely trainability — and the fix was not more resources but an architectural change that reshaped the loss surface gradient descent has to navigate. Modern refinements continue both threads: normalization placement, wider-vs-deeper trade-offs, depthwise-separable convolutions for efficiency, and eventually attention-based architectures. When you meet a new architecture, the productive question is the one this section asked of each generation: what failure mode does this design remove?

For your own work in this course, the practical defaults fall out directly: start from a small ResNet-style network rather than a plain deep stack; use ReLU-family activations, He initialization, and normalization; and if a deeper variant trains worse than a shallower one, suspect optimization before capacity.

Common pitfalls

Symptom: your deep network’s early layers never change; training loss barely moves. Cause: vanished gradients — saturating activations, depth without skips, or under-scaled initialization. Fix: check gradient norms per layer (as in this section’s first example) before touching hyperparameters; add skips or fix the initialization.
Symptom: adding layers makes training error worse, and dropout/weight decay do not help. Cause: the degradation problem — an optimization failure, not overfitting. Fix: residual connections; verify by confirming train (not just validation) error improved.
Symptom: shape mismatch (RuntimeError) at the out + x line of a residual block. Cause: the block changes spatial size or channel count, so the identity path no longer matches. Fix: pad the convs to preserve size, and use a 1x1 convolution (optionally strided) on the skip path when channels or resolution change — exactly what ResNet’s “projection shortcut” does.
Symptom: gradient norms print as exactly 0.00e+00. Cause: float32 underflow, either in the norm (the entries are tiny but nonzero, and .norm() squares them below the smallest positive float32) or, deeper still, in the entries themselves. Fix: nothing is recoverable at that point, since an update that small cannot change a float32 weight at any practical learning rate; redesign the network (skips, initialization, normalization) rather than raising the learning rate.
Symptom: a network with BatchNorm trains well but produces garbage at evaluation, or its outputs change with batch size. Cause: BN behaves differently in training mode (batch statistics) and eval mode (running averages); forgetting model.eval(), or evaluating with tiny batches while in training mode, mixes the two. Fix: call model.eval() before validation and model.train() before resuming; be suspicious of any train/eval gap that appears only when batch composition changes.

Try it yourself

Rerun the vanishing-gradient demo with nn.ReLU() in place of nn.Sigmoid(). Measure how the first-layer gradient norm scales with depth now, and reconcile what you see with the claim that ReLU “solves” vanishing gradients.
In the residual example, vary scale over 0.4, 0.5, 0.7, and 1.0 for both plain and residual stacks at 16 blocks. Which configuration is most sensitive to initialization scale, and in which direction can gradients explode?
Add a BatchNorm layer after each conv in Block (the real ResNet ordering is conv-BN-ReLU) and repeat the comparison at scale=0.5. Does normalization alone rescue the plain stack? Does it change the residual stack?
Build the smallest network you can that gets 100% training accuracy on the “+” vs “x” task from Section 2.3’s first example (after you have read it), with and without skips. At what depth does the plain version start losing to the residual one?

Before class you should be able to

Explain why AlexNet succeeded in 2012 in terms of the three obstacles LeNet hit (compute, data, trainability).
Argue why depth and parameter count are decoupled, using VGG-16 vs ResNet-152 as the example.
Explain mechanically why gradients vanish through deep sigmoid stacks, and explain why a float32 gradient norm can print as exactly zero while the gradient’s entries are not all zero.
Distinguish the degradation problem from overfitting — and say what evidence separates them (training error, not validation error).
Write the residual block equation and explain both its forward default (identity) and its backward effect.
Say what BatchNorm buys and why it alone was not enough beyond VGG depth.
Read a Block-with-skip implementation and spot the shape condition out + x requires.

Connects to

This session is covered by Quiz 2 (Module 2: S09–S12), given at S13.
HW2 (out at S12, due at S18) draws on the architectural judgment and gradient-norm-checking habit built here.
The residual connection returns around every attention and MLP sublayer in the Transformers of Module 3 — for exactly the reasons this section’s gradient table shows.

← S09 · Convolution · All readings · S11 · Transfer Learning and Data Augmentation →

How to read this

LeNet-5: the blueprint (1998)

AlexNet and the 2012 moment

The architecture zoo: depth is not parameters

Why depth was hard: the vanishing gradient, measured

Batch normalization: what it actually buys

The degradation experiment, reproduced in miniature

ResNet: learn the residual

After ResNet: scaling as a discipline

Going deeper

What to take from the history

Common pitfalls

Try it yourself

Further reading

Before class you should be able to

Connects to