Session S10
Why deep plain networks stopped training, and what one added line changed.
Deep Learning · CS 4715/5715 · University of Idaho
S09 ended with a small CNN inside the S08 training loop: two nn.Conv2d layers, each followed by ReLU and a \(2\times2\) max pool, then flatten and one nn.Linear.
Today's question: why was depth hard, and what fixed it?
LeCun, Bottou, Bengio and Haffner, after a decade of work at Bell Labs. The 1998 paper describes commercial deployment for reading checks.
The template was right in 1998. The history that follows removes these three obstacles.
Each block is one convolution's output after pooling, labeled \(C\times H\times W\): height grows with \(\sqrt{H}\), width with \(\log_2 C\), on the same scale as the AlexNet and VGG-16 drawings. Bars are vectors.
The next page's code builds these layers and prints each shape.
LeNet-5's layer sizes in PyTorch; each printed line is the shape after one layer.
import torch
import torch.nn as nn
lenet = nn.Sequential(
nn.Conv2d(1, 6, kernel_size=5), nn.Tanh(), nn.AvgPool2d(2),
nn.Conv2d(6, 16, kernel_size=5), nn.Tanh(), nn.AvgPool2d(2),
nn.Flatten(), nn.Linear(400, 120), nn.Tanh(),
nn.Linear(120, 84), nn.Tanh(), nn.Linear(84, 10))
h = torch.randn(1, 1, 32, 32) # one 32x32 grayscale digit
for layer in lenet:
h = layer(h)
if not isinstance(layer, nn.Tanh): # skip the activations
print(type(layer).__name__, tuple(h.shape))
print(sum(p.numel() for p in lenet.parameters()))
Conv2d (1, 6, 28, 28)
AvgPool2d (1, 6, 14, 14)
Conv2d (1, 16, 10, 10)
AvgPool2d (1, 16, 5, 5)
Flatten (1, 400)
Linear (1, 120)
Linear (1, 84)
Linear (1, 10)
61706
nn.Flatten(): \(16\cdot5\cdot5=400\) inputs.nn.AvgPool2d(2): the mean of each \(2\times2\) window (S09 took the maximum).The 1998 network had 60,000: its second convolution was only partly connected.
Two of LeNet's three obstacles went away at once.
Within about two years essentially every competitive entry was a deep CNN. 2012 is the usual start date of the deep learning era.
The same drawing and scale as LeNet-5's; each block is one convolution's output, pooled where the network pools.
A scaled LeNet: a PyTorch rebuild of the paper's layers has 60,965,224 parameters, 58.6M of them in the three fully connected layers.
With data and compute in place, the next thing to scale was depth.
| weights | \(C=64\) | |
|---|---|---|
| two \(3\times3\) layers | \(2\cdot9C^2=18C^2\) | 73,728 |
| one \(5\times5\) layer | \(25C^2\) | 102,400 |
The third obstacle, trainability, remained.
The same drawing and scale; each block is one stage's output after its \(2\times2\) max pool.
LeNet's template again: \(H\) and \(W\) shrink, the channels grow, and the head flattens the last map into fully connected layers.
ILSVRC (ImageNet Large Scale Visual Recognition Challenge) ran once a year. Top-5 error: the share of test images whose true class is not among the model's five highest-scoring classes.
Depth went from 8 layers (2012) to 152 (2015). Did the parameter count grow with it?
Reading 2.2's table: the papers' reported sizes; "~" marks a widely reported count rather than a number printed in the paper.
| network | venue, year | weight layers | parameters | signature idea |
|---|---|---|---|---|
| LeNet-5 | Proc. IEEE 1998 | 5 | ~60 K | conv + subsample template |
| AlexNet | NeurIPS 2012 | 8 | 60 M | ReLU, dropout, GPU training |
| VGG-16 | ICLR 2015 | 16 | 138 M | uniform \(3\times3\) stacks |
| GoogLeNet | CVPR 2015 | 22 | ~5 M | Inception modules; no big fully connected head |
| ResNet-152 | CVPR 2016 | 152 | ~60 M | residual connections |
| DenseNet-121 | CVPR 2017 | 121 | ~8 M | every layer sees all earlier features |
| EfficientNet-B0 | ICML 2019 | n/a | 5.3 M | depth, width, resolution scaled jointly |
VGG-16, at 16 layers, is more than twice the size of 152-layer ResNet-152: depth does not set the size. Where the parameters sit opens S11.
Back to the third obstacle, trainability: why does a deep plain stack not train? Take \(K\) Linear + sigmoid layers, numbered \(i=1,\dots,K\) from the input, on a batch \(X\) of shape \((N,32)\); the loss is computed from \(A_K\). \(\sigma'\) is the derivative of the sigmoid, \(\sigma'(z)=\sigma(z)(1-\sigma(z))\): the \(p(1-p)\) of S03.
\[A_0=X,\qquad Z_i=A_{i-1}W_i+b_i,\qquad A_i=\sigma(Z_i),\qquad W_i\in\mathbb R^{32\times32}\]The backward rule from S05, one layer at a time:
\[dZ_i=dA_i\odot\sigma'(Z_i),\qquad dA_{i-1}=dZ_i\,W_i^\top\]Layer \(i\)'s factor has two parts: \(\odot\,\sigma'(Z_i)\), then \(\cdot\,W_i^\top\). The first layer's gradient is a product of every factor above it; if the factors shrink the gradient, the product shrinks geometrically with depth.
S02's derivative table showed the maximum of \(\sigma'\), 0.25; here is why.
Result: every entry of \(\sigma'(Z_i)\) is at most 1/4, so row by row (lowercase \(da\) is one row of \(dA\)): \(\lVert da_{i-1}\rVert\le\tfrac14\,\lVert W_i\rVert\,\lVert da_i\rVert\) (\(\lVert W_i\rVert\): the most \(W_i^\top\) can stretch a vector, never more than the square root of the sum of its squared entries) (proved on the next page).
Each sigmoid layer can pass on at most a quarter of the gradient norm, times its weights' stretch.
Write \(W\) for \(W_i\). For row vectors \(v\), \(\lVert W\rVert=\max_{v\ne0}\lVert vW^\top\rVert/\lVert v\rVert\): the most \(W^\top\) can stretch a vector.
Self-check: for \(W=cI\) (\(32\times32\)), \(\lVert W\rVert=\lvert c\rvert\), while \(\sqrt{\sum_{j,k}W_{jk}^2}=\lvert c\rvert\sqrt{32}\): the bound holds but is not tight.
The previous page used this bound: it can be loose, but it is never violated.
The \(\tfrac14\) bound is the best case. Compute \(\sigma'(z)\) at five inputs (the dots), and compare ReLU's derivative.
z = torch.tensor([0.0, 1.0, 2.0, 3.0, 5.0])
s = torch.sigmoid(z)
print(s * (1 - s)) # sigma'(z) = s (1 - s)
tensor([0.2500, 0.1966, 0.1050, 0.0452, 0.0066])Hochreiter (1991) and Bengio, Simard and Frasconi (1994) analyzed this failure in recurrent networks; deep feedforward stacks inherit it.
Self-check. With \(\lVert W_i\rVert\le1\), what fraction of the gradient norm can survive 10 sigmoid layers at best? If every unit sits at \(z=2\) (\(\sigma'=0.1050\) in the printout), what fraction is left?
At best \((1/4)^{10}\approx9.5\times10^{-7}\); at \(z=2\), at most \(0.1050^{10}\approx1.6\times10^{-10}\): the typical case is far below the bound.
Build the stack at several depths (torch, nn as before); the loss is mean squared error against random targets, since only the gradient's size matters.
torch.manual_seed(0)
def make_plain_net(depth, width=32):
"""A deep stack of Linear + Sigmoid layers (early-1990s style)."""
layers = []
for _ in range(depth):
layers += [nn.Linear(width, width), nn.Sigmoid()]
layers += [nn.Linear(width, 1)]
return nn.Sequential(*layers)
x = torch.randn(64, 32) # N = 64 examples, 32 features
y = torch.randn(64, 1) # random regression targets
net[0]: the first nn.Linear(32, 32), weight \((32,32)\). net[-1]: the output layer, weight \((1,32)\)..weight.grad.norm() returns one number per weight matrix, \(\sqrt{\sum_{j,k}dW_{jk}^2}\), computed in float32.One backward pass per depth, same seed each time (make_plain_net, x, y from the previous page).
for depth in [2, 6, 12, 24, 48]:
torch.manual_seed(0)
net = make_plain_net(depth)
loss = nn.functional.mse_loss(net(x), y)
loss.backward()
g_first = net[0].weight.grad.norm().item() # first nn.Linear
g_last = net[-1].weight.grad.norm().item() # output nn.Linear
print(f"depth {depth:2d}: ||grad|| at output layer = {g_last:.2e}, "
f"at first layer = {g_first:.2e}, ratio = {g_first / g_last:.1e}")
depth 2: ||grad|| at output layer = 4.18e-01, at first layer = 1.68e-02, ratio = 4.0e-02
depth 6: ||grad|| at output layer = 1.48e+00, at first layer = 4.70e-06, ratio = 3.2e-06
depth 12: ||grad|| at output layer = 8.26e-01, at first layer = 2.05e-11, ratio = 2.5e-11
depth 24: ||grad|| at output layer = 7.76e-02, at first layer = 5.30e-21, ratio = 6.8e-20
depth 48: ||grad|| at output layer = 1.97e+00, at first layer = 0.00e+00, ratio = 0.0e+00
The output layer's gradient stays between 0.078 and 1.97 at every depth. The first layer's falls from \(1.7\times10^{-2}\) to a norm that prints 0.
Depth 12 from the table, now every weight gradient (make_plain_net, x, y as defined for the table).
torch.manual_seed(0)
net = make_plain_net(12)
nn.functional.mse_loss(net(x), y).backward()
norms = [m.weight.grad.norm().item()
for m in net if isinstance(m, nn.Linear)]
print(len(norms), f"{norms[0]:.2e}", f"{norms[-1]:.2e}")
ratios = [a / b for a, b in zip(norms, norms[1:])]
print(f"{min(ratios):.3f} {max(ratios):.3f}")
13 2.05e-11 8.26e-01
0.104 0.156
log scale; orange: the output layer
isinstance(m, nn.Linear) skips the nn.Sigmoid modules; zip(norms, norms[1:]) pairs each layer with the one above it.
A nearly constant factor per layer makes a straight descent on a log scale: printing per-layer gradient norms is how you diagnose a vanishing gradient.
Geometric decay: as depth doubles from 6 to 12 to 24, the table's ratio goes \(10^{-5.5}\to10^{-10.6}\to10^{-19.2}\): the exponent roughly doubles too. At depth 48 the first-layer norm prints 0; look at the gradient itself.
torch.manual_seed(0)
net = make_plain_net(48) # the depth-48 row again
nn.functional.mse_loss(net(x), y).backward()
g = net[0].weight.grad # float32, shape (32, 32)
print((g != 0).sum().item(), f"{g.abs().max().item():.2e}")
print(f"{g.norm().item():.2e} {g.double().norm().item():.2e}")
1020 2.09e-42
0.00e+00 1.44e-41
.norm() squares them, and \((2\times10^{-42})^2\) is below \(1.4\times10^{-45}\), the smallest positive float32.This is the vanishing gradient: early layers get almost no gradient and do not change.
2010–2015 brought partial fixes. AlexNet's ReLU has derivative 1 where active. S07 chose the weight variance \(2/D\) for a ReLU layer with \(D\) inputs (Kaiming/He, 2015). Rebuild the stack with ReLU, with and without that initialization.
def make_relu_net(depth, he, width=32):
layers = []
for _ in range(depth):
lin = nn.Linear(width, width)
if he:
nn.init.kaiming_normal_(lin.weight, nonlinearity="relu") # variance 2/D
nn.init.zeros_(lin.bias)
layers += [lin, nn.ReLU()]
return nn.Sequential(*layers, nn.Linear(width, 1))
| weight entries drawn from | standard deviation, \(D=32\) | |
|---|---|---|
nn.Linear default | uniform on \([-1/\sqrt{32},\,1/\sqrt{32}]\) | \(1/\sqrt{96}\approx0.102\) |
kaiming_normal_ | normal, variance \(2/D\) | \(\sqrt{2/32}=0.25\) |
nn.init.kaiming_normal_ redraws lin.weight in place (the trailing underscore marks in-place functions); nn.init.zeros_ sets the bias to 0.
The sigmoid table's loop: seed 0, the same x and y (make_relu_net from the previous page).
print("depth default He init")
for depth in [2, 6, 12, 24, 48]:
g = []
for he in [False, True]:
torch.manual_seed(0)
net = make_relu_net(depth, he)
nn.functional.mse_loss(net(x), y).backward()
g.append(net[0].weight.grad.norm().item())
print(f"{depth:5d} {g[0]:.2e} {g[1]:.2e}")
depth default He init
2 2.21e-01 3.97e-01
6 2.70e-03 1.32e+00
12 7.39e-06 4.16e-01
24 4.11e-11 1.14e+00
48 7.38e-21 4.94e-02
The gradient now reaches the first layer, yet deeper plain networks still trained worse.
The other partial fix, BatchNorm (BN, S07). Reading 2.2: a 30-block Linear + ReLU stack, width 64, every weight matrix multiplied by 0.5, a batch of shape \((256,64)\), loss = mean of the squared output. Printed values, tabulated.
| activation standard deviation after block | first-layer gradient norm | ||||
|---|---|---|---|---|---|
| 1 | 10 | 20 | 30 | ||
| without BN | 0.17 | 0.045 | 0.041 | 0.041 | 0 |
| with BN | 0.58 | 0.61 | 0.61 | 0.60 | 32 |
nn.BatchNorm1d(64) on \((256,64)\): per feature, subtract the mean and divide by the standard deviation over the 256 rows (dim=0), then apply a learned scale \(\gamma\) and shift \(\beta\), 64 values each.BN fixes the scale of the signal. Deep plain networks can still fit their training data worse.
With BatchNorm throughout, He et al. (the ResNet paper) trained on CIFAR-10, 60,000 color images of \(32\times32\) pixels in 10 classes, and found that a 56-layer plain network had higher training error than a 20-layer one.
| training error | validation error | what failed | what helps | |
|---|---|---|---|---|
| Overfitting | low | high | the model fits noise in the training set | more data, regularization |
| Degradation | higher for the deeper net | higher | the optimizer does not fit the data | an easier-to-optimize architecture |
The diagnosis needs the training error. Regularization cannot help a network that does not fit its training set.
Reading 2.2 compares plain and residual training fit at toy scale, on a task one block can fit (torch as before).
import numpy as np
def make_data(n, rng):
X = rng.normal(0.0, 0.5, size=(n, 2)).astype(np.float32)
y = (X[:, 0] * X[:, 1] > 0).astype(np.int64)
X += rng.normal(0.0, 0.1, size=X.shape).astype(np.float32)
return torch.from_numpy(X), torch.from_numpy(y)
X, y = make_data(256, np.random.default_rng(0))
print(X.dtype, y.dtype, y.float().mean().item())
torch.float32 torch.int64 0.47265625blue circles: class 1; orange squares: class 0
float32 inputs, int64 labels: what nn.CrossEntropyLoss expects.Class 1 is 47.3% of this seed's points, so always answering class 0 scores 0.527.
Reading 2.2's block: two nn.Linear(32, 32) layers; skip switches on one added line.
class Block(nn.Module):
def __init__(self, width, skip):
super().__init__()
self.fc1 = nn.Linear(width, width)
self.fc2 = nn.Linear(width, width)
self.relu = nn.ReLU()
self.skip = skip
def forward(self, x):
out = self.fc2(self.relu(self.fc1(x)))
if self.skip:
out = out + x
return self.relu(out)
def make_net(n_blocks, skip, width=32):
return nn.Sequential(nn.Linear(2, width), nn.ReLU(),
*[Block(width, skip) for _ in range(n_blocks)],
nn.Linear(width, 2))
Linear(2, 32), ReLUBlock(32, skip)Linear(32, 2)*[...] unpacks \(n\) new blocks; plain and residual nets differ only in skip.
make_data and make_net from the previous two pages. 600 SGD steps on minibatches of 64, then accuracy on all 256 training points.
def train_accuracy(n_blocks, skip, seed, steps=600):
rng = np.random.default_rng(seed)
torch.manual_seed(seed)
X, y = make_data(256, rng)
net = make_net(n_blocks, skip)
opt = torch.optim.SGD(net.parameters(), lr=0.05, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()
for _ in range(steps):
idx = torch.randint(0, len(X), (64,)) # 64 random row indices
opt.zero_grad()
loss_fn(net(X[idx]), y[idx]).backward()
opt.step()
with torch.no_grad():
return (net(X).argmax(1) == y).float().mean().item()
X[idx]: a \((64,2)\) minibatch. The loop is S08's: zero, loss, backward, step.argmax(1): the larger logit in each row; .float().mean() of the matches is the fraction correct.skip.Mean training accuracy over seeds 0, 1, 2 (train_accuracy from the previous page).
print("blocks plain residual")
for n in [1, 4, 16, 32]:
row = [np.mean([train_accuracy(n, skip, s) for s in range(3)])
for skip in [False, True]]
print(f"{n:6d} {row[0]:.4f} {row[1]:.4f}")
blocks plain residual
1 0.8815 0.8854
4 0.5091 0.8841
16 0.5221 0.8906
32 0.5221 0.8880
net(X).argmax(1).unique() has one entry), so its accuracy is that class's share: 0.527, 0.520, 0.520 for class 0; at 4 blocks seed 1 chose class 1 (0.480).More depth, same data and budget, worse training fit; the residual column differs by one line, out = out + x. Degradation, or the vanishing gradient?
The miniature uses default initialization and no BatchNorm, the setting where the ReLU stack's gradient vanished. "+ BN": nn.BatchNorm1d(32) after each nn.Linear in every block.
| blocks | plain | plain + BN | residual + BN |
|---|---|---|---|
| 1 | \(7.8\times10^{-3}\) | 0.36 | 0.45 |
| 4 | \(2.7\times10^{-5}\) | 0.40 | 1.2 |
| 16 | \(2.2\times10^{-14}\) | 55 | 2.6 |
| 32 | prints 0 | \(2.8\times10^{4}\) | 9.2 |
Separate runs, tabulated: first-layer gradient norm of the full-batch loss before training, seed 0.
BatchNorm turns the plain net's vanishing gradient into an exploding one.
Training accuracy, mean of seeds 0–2, the same recipe; "+ BN" nets evaluated after net.eval().
| blocks | plain | plain + BN | residual | residual + BN |
|---|---|---|---|---|
| 1 | 0.8815 | 0.8919 | 0.8854 | 0.8906 |
| 4 | 0.5091 | 0.8659 | 0.8841 | 0.8815 |
| 16 | 0.5221 | 0.7539 | 0.8906 | 0.8698 |
| 32 | 0.5221 | 0.5417 | 0.8880 | 0.8841 |
The one added line, out = out + x, fits at every depth. A deeper plain net that fits worse with a healthy gradient is the paper's finding; this miniature cannot isolate it.
He, Zhang, Ren and Sun (CVPR 2016): the sum \(Y\) (out in code) is branch plus input; a ReLU follows.
The branch only has to learn the difference \(F(X)=Y-X\).
Block input \(X\), output \(Y\), upstream gradient \(dY=\partial L/\partial Y\) (S05). The block's partial derivatives \(\partial F_j/\partial X_k\), for every output \(j\) and input \(k\), form a matrix, the Jacobian \(J_F\) (reading 1.4; \(X\) flattened to a vector). One symbol for the whole block lets us add \(I\) and multiply blocks (slide 33). Example: for a Linear layer \(F(X)=XW\), \(J_F=W^\top\), so \(dY\,J_F=dY\,W^\top\), S05's rule.
Result: the residual block has one extra term, \(dY\), passed through unchanged.
A small branch on a 3-vector (torch as before): \(F(x)=\tanh(xW)\) with a fixed \(3\times3\) matrix \(W\). torch.autograd.functional.jacobian(f, x) returns the matrix of \(\partial f_i/\partial x_j\).
torch.manual_seed(0)
W = torch.randn(3, 3) * 0.5
f = lambda v: torch.tanh(v @ W) # the branch F
x = torch.randn(3)
J_F = torch.autograd.functional.jacobian(f, x)
J_Y = torch.autograd.functional.jacobian(lambda v: f(v) + v, x)
print(J_F)
print(J_Y)
print(torch.allclose(J_Y, J_F + torch.eye(3)))
tensor([[ 0.6367, 0.2349, 0.1667],
[-0.1197, -0.4423, 0.3418],
[-0.6162, -0.3955, -0.2034]])
tensor([[ 1.6367, 0.2349, 0.1667],
[-0.1197, 0.5577, 0.3418],
[-0.6162, -0.3955, 0.7966]])
True
torch.eye(3) is the \(3\times3\) identity \(I\).The identity term is exact: the skip adds \(I\) to the Jacobian, whatever the branch computes.
Block \(k\) has Jacobian \(J_k+I\). Chain two blocks: \(dX_0=dX_2\,(J_2+I)(J_1+I)\).
\[(J_2+I)(J_1+I)=J_2J_1+J_2+J_1+I\]| term in \(dX_0\) | block 2 | block 1 | weight factors |
|---|---|---|---|
| \(dX_2\,J_2J_1\) | branch | branch | 2 |
| \(dX_2\,J_2\) | branch | skip | 1 |
| \(dX_2\,J_1\) | skip | branch | 1 |
| \(dX_2\) | skip | skip | 0 |
\(n\) blocks: \(2^n\) terms, one per choice of branch or skip at each block. Exactly one, \(dX_n\) itself, has no weight factor.
Veit et al. (NeurIPS 2016) studied these paths; the next three optional pages go through what they found. A path's length is the number of branches it takes (the last column above).
A plain stack has one path, through every weight. A residual stack adds paths of every length.
Veit et al. substitute \(X_k=X_{k-1}+F_k(X_{k-1})\) block by block (their eqs. 4–6); for \(n=3\):
\[\begin{aligned}X_3&=X_2+F_3(X_2)=X_1+F_2(X_1)+F_3\big(X_1{+}F_2(X_1)\big)\\&=X_0+F_1(X_0)+F_2\big(X_0{+}F_1(X_0)\big)+F_3\big(X_0{+}F_1(X_0){+}F_2(X_0{+}F_1(X_0))\big)\end{aligned}\]A path is a code \(b\in\{0,1\}^n\): at block \(k\) it takes the branch \(F_k\) (\(b_k=1\)) or the skip (\(b_k=0\)). Its length \(x=b_1+\dots+b_n\) counts its branches; \(\binom nx\) paths have length \(x\), for \(n=3\): 1, 3, 3, 1. A plain stack has one path, of length \(n\).
from math import comb
n = 54 # 54 modules: the 110-layer ResNet of Veit et al.
counts = [comb(n, x) for x in range(n + 1)] # paths of length x
print(sum(counts), sum(x * c for x, c in enumerate(counts)) / 2**n)
print(f"{sum(counts[19:36]) / 2**n:.3f}") # lengths 19 to 35
18014398509481984 27.0
0.980
With 54 blocks, 0.980 of the paths take 19 to 35 branches (Veit et al.: more than 95%).
Delete block \(k\) of a trained network: drop its branch, keep its skip (\(X_k=X_{k-1}\)). A path remains if it takes none of the deleted branches.
After \(d\) of \(n\) blocks are deleted, a fraction \(\binom{n-d}{x}/\binom nx\) of the paths of length \(x\) remain (Veit et al., eq. 8); for \(d=1\), \((n-x)/n\). All lengths together: \(2^{n-1}\) of \(2^n\), half.
from math import comb
n, d = 54, 1 # delete 1 of 54 blocks
lengths = [0, 5, 17, 27, 35]
print([round(comb(n - d, x) / comb(n, x), 3) for x in lengths])
[1.0, 0.907, 0.685, 0.5, 0.352]
Veit et al. §4, CIFAR-10: deleting any one block of their trained 110-layer ResNet barely changed its error (a downsampling block raised it a little); deleting any one layer of a 15-layer VGG dropped its accuracy to chance. Deleting more blocks, or shuffling their order, raised the ResNet's error smoothly.
The paths behave like an ensemble, in Veit et al.'s sense: they do not strongly depend on each other.
Each branch a path takes adds one factor \(J_k\) to its gradient, as each layer did in the plain stack; if the factors shrink it, a path's gradient falls geometrically with its length. Veit et al. §5 measured it (54 modules, CIFAR-10):
from math import comb
n = 54
print(f"{sum(comb(n, x) for x in range(5, 18)) / 2**n:.4f}") # lengths 5 to 17
0.0045
A ResNet does not carry its gradient through the whole depth; its short paths carry it. The long paths still vanish: the identity term does not guarantee a large gradient (next page).
Two conditions sit behind "passes the gradient through".
Self-check. Suppose a branch has learned \(J_F=-0.9\,I\). What does one block multiply the gradient by, and what is left after 10 such blocks?
\(J_F+I=0.1\,I\): each block multiplies by 0.1, ten blocks by \(0.1^{10}=10^{-10}\). Across many blocks the product can still shrink, explode, or cancel; the identity term improves the path.
Reading 2.2's convolutional block (torch, nn as before). This Block uses nn.Conv2d; it replaces the nn.Linear one of the miniature. skip switches the one added line.
class Block(nn.Module):
"""Two 3x3 convs. With skip=True this is a basic ResNet block."""
def __init__(self, channels, skip):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
self.relu = nn.ReLU()
self.skip = skip
def forward(self, x): # x: (4, 8, 8, 8)
out = self.conv2(self.relu(self.conv1(x))) # F(x): (4, 8, 8, 8)
if self.skip:
out = out + x # the residual connection
return self.relu(out)
padding=1 keeps \(8\times8\) and channels stays 8, so out + x adds two tensors of shape \((4,8,8,8)\).
Stack \(n\) copies of Block from the previous page. init="half" halves every conv weight, so each block shrinks its input (pre-2015 under-scaling); init="he" redraws them with Kaiming/He initialization, as on the ReLU page (biases keep their default here).
def grad_at_first_layer(n_blocks, skip, init):
torch.manual_seed(0)
blocks = nn.Sequential(*[Block(8, skip) for _ in range(n_blocks)])
with torch.no_grad(): # edit weights without recording
for p in blocks.parameters():
if p.dim() == 4 and init == "he": # conv weights, (8, 8, 3, 3)
nn.init.kaiming_normal_(p, nonlinearity="relu")
elif p.dim() == 4: # init == "half"
p.mul_(0.5) # in place: halve every entry
blocks(torch.randn(4, 8, 8, 8)).pow(2).mean().backward() # x: (N, C, H, W)
return blocks[0].conv1.weight.grad.norm().item()
print("blocks plain 0.5 resid 0.5 plain He resid He")
for n in [2, 8, 16, 32]:
g = [grad_at_first_layer(n, s, init) for init in ["half", "he"]
for s in [False, True]]
print(f"{n:6d}" + "".join(f" {v:10.2e}" for v in g))
Within each initialization, same seed, weights and input: only out + x differs.
The previous page's printout: the first conv layer's gradient norm.
blocks plain 0.5 resid 0.5 plain He resid He
2 2.44e-05 5.23e-02 1.83e-01 1.00e+00
8 4.39e-14 5.86e-02 3.54e-01 3.73e+02
16 0.00e+00 5.42e-02 3.11e-01 1.52e+05
32 0.00e+00 6.36e-02 5.96e-05 1.79e+08
One added line, out + x, kept the path open under both initializations; it does not control the gradient's size. Real ResNet blocks add BatchNorm for that (S11 builds one).
Depth became usable once each block could default to passing its input through.
Defaults: a small ResNet-style network, ReLU-family activations, Kaiming/He initialization, normalization.
| symptom | cause | check or fix |
|---|---|---|
| early layers never change; the loss barely moves | vanishing gradient | print per-layer gradient norms; add skips, fix the initialization |
| gradient norms grow toward the input | exploding gradient | per-layer gradient norms; add skips and normalization |
| more layers, worse training error; dropout and weight decay do not help | degradation | compare training error; add residual connections |
a gradient norm prints exactly 0.00e+00 | float32 underflow of the norm's squares, or of the entries | redesign: skips, initialization, normalization; no learning rate helps |
| a BatchNorm net evaluates badly, or changes with batch size | training mode at evaluation | model.eval() before validation, model.train() before training |
If a deeper model trains worse than a shallower one, suspect optimization before capacity.
A 20-block plain CNN reaches 70% training accuracy; an 8-block version reaches 90% training accuracy on the same data. Overfitting or degradation? What do you measure first, and what do you change?
Degradation: the deeper model fits its training data worse (overfitting would show lower training error). Print per-layer gradient norms first, where vanishing or exploding shows, then add residual connections; neither regularization nor BatchNorm alone fixes it.