Session S15

Guided Review: S01–S13

Training a network takes five things: a model, a loss, the loss's gradient, a step rule, and a loop that runs them. Which part of S01–S13 built each one, which architecture carries it for images and for sequences, and where does each part break?

A self-study review deck: S15 is not held this term.

Deep Learning · CS 4715/5715 · University of Idaho

How to use this deck: eight blocks, each ending in a self-check

The title page asks one question about five parts. The deck answers it in eight blocks, in the order S01–S13 built the parts.

  • Blocks, in course order: S01–S02, S03, S04–S05, S06–S07, S08, S09–S10, S11–S12, S13.
  • In each block: the rules, in the source session's notation; a worked example to check on the page; a self-check. Answer it first: the answer opens on click.
  • Every number is derived on its page or copied, with its source, from an earlier session's run.
  • The S15 review guide (the S15 reading) links each block to its readings; the deck ends with its checklist.
  • Still assessed: S13 is on Quiz 3 (at S23); the Module 1–2 skills carry HW2 (due at S18) and HW3 (out at S18).

Start with the map: the five parts, and the sessions that built each.

The map: one training cycle in five parts, and the sessions that built them

Training repeats one cycle: the model predicts, the loss scores the prediction, backpropagation returns the gradient, the step rule moves the parameters \(\theta\), and the loop starts again on the next minibatch.

repeat for every minibatch model f(x; θ) S01–S02 · block 1 loss L = −log py S03 · block 2 gradient ∇θ L, by backprop S04–S05 · block 3 step rule SGD, momentum, Adam S03, S06 · block 4 the loop in PyTorch S08 · block 5 images (grids) convolution, residual blocks S09–S10 · block 6 sequences recurrence, gates (RNN, LSTM) S13 · block 8 keep it working initialization, normalization, regularization S06–S07 · block 4 training practice transfer, augmentation, debugging, seeds S11–S12 · block 7

Top row: the cycle. Bottom row: a model for each kind of data, and the tools that keep the cycle working. Each block opens by naming its box.

Block 1 starts at the first box: what the model must contain.

Block 1 · S01–S02: stacked linear layers collapse into one

Block 1 is the map's first box, the model: S02's multilayer perceptron (MLP). S02's finding: depth alone adds nothing.

\(\mathbf{x}\) has \(D\) entries, \(W_1\) is \(H\times D\), \(W_2\) is \(K\times H\). Substitute the first layer into the second:

\[\hat{\mathbf{y}}=W_2(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2=\underbrace{(W_2W_1)}_{K\times D}\,\mathbf{x}+\underbrace{(W_2\mathbf{b}_1+\mathbf{b}_2)}_{K\text{ entries}}\]
  • The result is one linear layer; layer by layer, a stack of any depth collapses the same way. Reading 1.2's ten-layer stack, 4 inputs to 3 outputs, prints:

parameters in the stack: 2307, in the collapsed layer: 15
  • The cure (S02): a nonlinear activation \(f\) between the layers, \(\mathbf{h}=f(W_1\mathbf{x}+\mathbf{b}_1)\).

A forgotten activation raises no error: the model just performs like a linear one.

The sigmoid for a binary output, ReLU for hidden layers

Which nonlinearity? S02 compared the sigmoid, tanh and ReLU. The table gives the two default jobs; tanh returns in block 8, inside S13's recurrence.

formulalocal derivativejob
sigmoid\(\sigma(z)=1/(1+e^{-z})\)\(\sigma(z)\,(1-\sigma(z))\le1/4\)binary output unit (S03)
ReLU\(\operatorname{ReLU}(z)=\max(0,z)\)1 for \(z>0\), 0 for \(z<0\)default hidden unit
  • Backprop multiplies one local derivative per layer: reading 1.2's 20 sigmoid factors give \(5.77\times10^{-17}\), a vanishing gradient; tanh's is \(1-\tanh^2(z)\le1\).
  • Dead ReLU: a unit whose pre-activation is negative for every training input gets zero gradient and never learns (causes: too large a learning rate, a bad initialization).
  • Each hidden unit is a linear classifier with its own hyperplane, \(\mathbf{w}_j^\top\mathbf{x}+b_j=0\); the output layer is a linear model over the units' outputs.

Next page: two hidden units, two hyperplanes, enough for exclusive or (XOR).

Worked: two hidden ReLU units compute XOR

S02: one unit fires for "both inputs high", one for "both low", and the output fires when neither does. On reading 1.2's points \((\pm1,\pm1)\), XOR is \(+1\) when the signs differ.

\(h_1=\operatorname{ReLU}(x_1+x_2-1)\), \(h_2=\operatorname{ReLU}(-x_1-x_2-1)\), \(s=1-2h_1-2h_2\):

\((x_1,x_2)\)\(h_1\)\(h_2\)\(s\)label
\((1,1)\)10−1−1
\((-1,-1)\)01−1−1
\((1,-1)\)001+1
\((-1,1)\)001+1

Self-check. Why does a ten-layer network with no activations still fail XOR?

It collapses to one score \(s=w_1x_1+w_2x_2+b\), a line. XOR needs \(s(1,-1)+s(-1,1)=2b>0\) and \(s(1,1)+s(-1,-1)=2b<0\): impossible at any depth.

Block 2: the model can represent XOR; now a loss must say how wrong it is.

Block 2 · S03: cross-entropy's push survives a confident error

Block 2 is the map's second box, the loss. Regression uses the mean squared error (MSE); classification uses cross-entropy, \(L=-\log p_y\) for the true class \(y\). Why not MSE for both? S03: a sigmoid head \(p=\sigma(z)\), label \(y=1\).

loss\(L\)\(\partial L/\partial z\)
MSE\((p-y)^2\)\(2(p-y)\,p\,(1-p)\)
cross-entropy\(-\log p\)\(p-y\)
 

z (logit)   p=sigmoid(z)   |dL/dz| MSE   |dL/dz| CE
  -8.0         0.0003      0.000670      0.9997
  • Reading 1.3's first row: at \(p=0.0003\), MSE's gradient, 0.000670, is about 1,500 times smaller than 0.9997. It carries \(p(1-p)\), the sigmoid's derivative, which vanishes at saturation; cross-entropy's log cancels it.
  • Softmax with cross-entropy: \(\nabla_{\mathbf{z}}L=\mathbf{p}-\operatorname{onehot}(y)\) (S03).

Worked: subtract the largest logit before exponentiating

Cross-entropy needs \(p_y\) from the softmax, \(p_k=e^{z_k}/\sum_je^{z_j}\). S03's worked number: \(\mathbf{z}=(2,1,0)\) gives \(\mathbf{p}\approx(0.665,0.245,0.090)\). Add 1000 to every logit:

  1. \(\mathbf{z}=(1002,1001,1000)\): each \(e^{z_k}\) overflows (float64: past \(e^{709}\), S03); \(\infty/\infty\) is nan.
  2. \(\operatorname{softmax}(\mathbf{z}+c)=\operatorname{softmax}(\mathbf{z})\), since \(e^{c}\) cancels; subtract \(m=1002\).
  3. \((0,-1,-2)\): \(e^{0},e^{-1},e^{-2}=1,\,0.368,\,0.135\), sum 1.503, \(\mathbf{p}\approx(0.665,0.245,0.090)\).
  • Underflow (S03): even after the shift, logits \((15,1000,-8)\) give \(p_0=e^{-985}\), which rounds to 0, so \(-\log p_0=\infty\). Computing \(\log p_0=z_0-m-\log\sum_je^{z_j-m}\) directly, with \(m=1000\), gives the loss 985.0000 (reading 1.3).

That fused log-softmax is why nn.CrossEntropyLoss takes raw logits.

Self-check: the same overflow in float32

Block 2's check: the previous page's overflow, in PyTorch's default dtype, float32 (S08). Its largest number is about \(3.40\times10^{38}\approx e^{88.72}\), so \(e^{z}\) overflows just past \(z=88.72\) (float64: 709.78).

Float32 logits \(\mathbf{z}=(100,90,80)\), true class 2. (a) What does the naive softmax return? (b) After subtracting the largest logit, what are \(\mathbf{p}\) and \(-\log p_2\)? (c) Would the naive softmax fail on these logits in float64?

(a) \(e^{100}\) and \(e^{90}\) overflow, so the sum is \(\infty\): (nan, nan, 0). (b) \((0,-10,-20)\): \(\mathbf{p}\approx(0.99995,\,4.54\times10^{-5},\,2.06\times10^{-9})\), and \(-\log p_2=20+\log(1+e^{-10}+e^{-20})\approx20.0000\). (c) No: \(100<709.78\), so in float64 the naive softmax returns (b)'s \(\mathbf{p}\).

Block 3: the loss is one number; training needs its gradient for every parameter.

Block 3 · S04–S05: backprop multiplies local derivatives

Block 3 is the map's third box, the gradient. S04's warm-up graph: \(a=wx\), \(b=a+c\), \(L=b^2\), at \(w=2\), \(x=3\), \(c=1\). Each operation knows only its own local derivative.

w = 2x = 3a = 6c = 1b = 7L = 49 ×+square 422814141 14 in each blue box: the forward value orange: ∂L/∂(node), filled in backward from L
\[\frac{\partial L}{\partial w}=\underbrace{2b}_{14}\cdot\underbrace{\partial b/\partial a}_{1}\cdot\underbrace{\partial a/\partial w}_{x\,=\,3}=42\]

Likewise \(\partial L/\partial x=14\cdot1\cdot w=28\), \(\partial L/\partial c=14\cdot1=14\). S04's local-derivative table: a multiply slot's derivative is the other slot; an add's, 1.

\(\partial L/\partial b=14\) is computed once and reused by every path below.

A node that feeds two operations gets the sum of both gradients

Each warm-up node fed one operation. When a node \(a\) feeds \(b_1,\dots,b_k\), S04's reuse rule adds the paths (S05):

\[\frac{\partial L}{\partial a}=\sum_{i=1}^{k}\frac{\partial L}{\partial b_i}\,\frac{\partial b_i}{\partial a}\]
  • \(L=x\cdot x\): each slot gives \(x\), two paths: \(\partial L/\partial x=x+x=2x\).
  • HW1's test, walked in S05: \(f=a\cdot a+a\) at \(a=3\). The add deposits 1, each multiply slot deposits 3: a.grad \(=1+3+3=7=2a+1\). With grad = instead of grad +=, an engine reports 4 or 6 (S04).
  • For a batch layer \(Z=XW+b\) (S05), \(dZ=\partial L/\partial Z\): \(dW=X^\top dZ\), \(dX=dZ\,W^\top\), \(db=\sum_n dZ_{n,:}\) since every row reuses \(b\); each has its variable's shape.
  • .grad holds \(\partial L/\partial(\text{node})\) (S04). PyTorch keeps it on leaf tensors such as parameters and adds into it on every backward call (reading 1.5); block 5 returns to this.

Reverse mode: one backward sweep gives every gradient

Why sweep backward from \(L\), not forward from each input? S04 compared the two directions of the chain rule.

modeone sweep carriessweeps for \(n\) parameters, one loss
forwardhow every node changes per unit change of one input\(n\), one per input
reversehow \(L\) changes per unit change of each node1, all \(\partial L/\partial(\text{node})\) at once
  • Forward mode needs one sweep per input, so with millions of parameters and one loss, reverse mode wins by a factor of millions. Backprop is reverse-mode automatic differentiation.
  • Cost (S04): the backward pass costs about two forward passes, so a training step costs about three, not a million.
  • Memory (S04): backward factors are stored forward values, so all activations stay in memory until the backward pass ends.

Nudging each parameter costs two forward passes (S04): fine as a test, next page.

The central-difference check confirms backprop's 42

Backprop bugs give wrong numbers, not errors. HW1's tests ran S04's central difference, \(\big(L(p+\varepsilon)-L(p-\varepsilon)\big)/(2\varepsilon)\), error \(O(\varepsilon^2)\), against your engine; reading 1.4 shows how to write one.

The warm-up graph as a function of \(w\) (here \(p=w\)) is \(L(w)=(3w+1)^2\). With \(\varepsilon=10^{-3}\): \((7.003^2-6.997^2)/0.002=(49.042009-48.958009)/0.002=42.000\).

Self-check. Change the add to \(b=a+w\), so \(w\) feeds both operations (\(w=2\), \(x=3\), \(L=b^2\)). What is \(\partial L/\partial w\)? What does an engine with grad = report?

\(b=8\), \(\partial L/\partial b=16\); via the multiply \(16x=48\), via the add 16: sum 64 (check: \(L=16w^2\), \(L'=32w=64\)). Overwriting keeps 48 or 16; the central difference gives 64.000.

Block 4: the gradient points uphill; the step rule decides how far to move.

Block 4 · S03: each step multiplies \(w\) by \(1-\eta a\)

Block 4 is the map's fourth box, the step rule. Its plainest form is S03's \(\theta\leftarrow\theta-\eta\,\nabla_\theta L\); how large may \(\eta\) be? S03's bound, on a quadratic with curvature \(a\):

  1. \(L=(a/2)\,w^2\), \(L'=a\,w\): one step gives \(w\leftarrow(1-\eta a)\,w\), so \(w_t=(1-\eta a)^t\,w_0\).
  2. It converges exactly when \(\lvert1-\eta a\rvert<1\): \(0<\eta<2/a\). If \(\eta a>1\), \(w\) alternates sign.

S03's bowl \((w_0^2+25\,w_1^2)/2\) from \((10,1)\): curvatures 1 and 25, so \(\eta<2/25=0.08\).

\(\eta\)\(1-25\eta\)\(1-\eta\)regimeloss at step 100 (reading 1.3)
0.0050.8750.995crawling18.35
0.06−0.50.94converging0.0002111
0.081−1.0250.919exploding1745

The steepest direction caps \(\eta\), so the shallow one crawls; momentum, next page, builds speed along it.

S06: momentum turns a steady \(g\) into \(g/(1-\beta)\)

On minibatches the previous page's rule is stochastic gradient descent (SGD), \(W\leftarrow W-\eta G\), and it still crawls along the bowl's shallow direction. S06's momentum keeps a memory of past gradients:

\[V_t=\beta V_{t-1}+G_t,\quad V_0=0,\qquad W\leftarrow W-\eta V_t,\qquad\beta=0.9\]
  1. Constant \(g\): \(v_t=g\,(1+\beta+\cdots+\beta^{t-1})=g\,(1-\beta^t)/(1-\beta)\to g/(1-\beta)=10g\).
  2. Alternating \(\pm g\): once \(v\) flips sign every step with size \(u\), \(-\beta u+g=u\), so \(u=g/(1+\beta)\approx0.53\,g\).
  • A steady-gradient result, not a bound relative to the current gradient (reading 1.6): after a sign change, the old velocity can still overshoot (S06).

Gradients that keep their sign build speed (10×); alternating ones partly cancel.

Adam steps about \(\eta\) per coordinate; AdamW decays outside it

Momentum remembers direction. Adam (S06) also divides each coordinate by its recent gradient size:

\[\begin{aligned}M_t&=0.9\,M_{t-1}+0.1\,G_t, &\qquad \widehat M_t&=M_t/(1-0.9^t),\\ V_t&=0.999\,V_{t-1}+0.001\,G_t\odot G_t, &\qquad \widehat V_t&=V_t/(1-0.999^t),\\ W&\leftarrow W-\eta\,\widehat M_t\oslash\big(\sqrt{\widehat V_t}+\epsilon\big).\end{aligned}\]
  • Steady \(g\): \(\widehat M_t=g\), \(\widehat V_t=g^2\), so the step is \(\eta\,g/(\lvert g\rvert+\epsilon)\approx\eta\) (\(\epsilon\) is tiny) at any scale: an approximation, not a cap. Its \(V\) averages squared gradients; momentum's was a velocity.
  • Weight decay's penalty \((\lambda/2)\lVert W\rVert_F^2\) has gradient \(\lambda W\). torch.optim.Adam(weight_decay=...) adds it to \(G\) before the moments (coupled L2), so \(\sqrt{\widehat V}\) divides it too: not a uniform shrink.
  • torch.optim.AdamW applies \(W\leftarrow(1-\eta\lambda)\,W\) directly, outside the moments.

Initialization: each layer should multiply \(\mathbb{E}[x^2]\) by about 1

Now the map's "keep it working" box: a deep stack must pass its signal forward at a usable scale, and the initialization (S07) sets that scale before step 1. Write \(q=\mathbb{E}[x^2]\) for a layer's average squared activation.

  • Fan-in \(D\), weight variance \(\sigma^2\): a linear layer multiplies \(q\) by \(D\sigma^2\), and a ReLU halves it, so \(q_{\text{next}}\approx(D\sigma^2/2)\,q\). On the std, a factor 0.5 is \(\sqrt{0.5}\approx0.707\) (S07): \(0.707^{39}\approx10^{-6}\).
  • Xavier at equal widths, \(\sigma^2=1/D\), suits tanh-like units; under ReLU its factor is \(1/2\). Kaiming, \(\sigma^2=2/D\), undoes ReLU's \(1/2\).

S07's 40-layer ReLU run, width 256 (reading 1.7, two of three rows; \(n\) is \(D\)):


activation std, layer:           1           5          10          20          40
     Xavier  sqrt(1/n)    5.87e-01    1.51e-01    2.57e-02    6.90e-04    5.96e-07
     Kaiming sqrt(2/n)    8.30e-01    8.54e-01    8.21e-01    7.07e-01    6.25e-01

Under ReLU, Xavier's activation std decays to \(6\times10^{-7}\); Kaiming's stays above 0.6.

Normalization chooses an axis; regularization limits overfitting

Initialization sets the scale at step 0; normalization layers re-standardize activations during training. S07's example: \(X=\left[\begin{smallmatrix}1&2\\3&6\end{smallmatrix}\right]\), rows are samples, columns features.

layermean and variance over\(X\) becomesin evaluation
BatchNorm1deach column (dim=0): \(\mu_B=[2,4]\), \(s_B^2=[1,4]\)\(\approx\left[\begin{smallmatrix}-1&-1\\1&1\end{smallmatrix}\right]\)stored running statistics
LayerNormeach row (dim=-1): \(\mu_L=[1.5,4.5]^\top\), \(s_L^2=[0.25,2.25]^\top\)\(\approx\left[\begin{smallmatrix}-1&1\\-1&1\end{smallmatrix}\right]\)the same computation
  • BatchNorm needs batch statistics, so small batches hurt; LayerNorm uses none.
  • Regularizers (S06–S07): weight decay; dropout, \((X\odot M)/(1-p)\) in training with drop probability \(p\) and a random 0/1 mask \(M\), \(X\) in evaluation; early stopping at the best validation checkpoint (S07's no-decay run: validation MSE 0.0275 at best, 0.2335 at the end).
  • Reading 1.7's number-one regularization bug: forgetting model.eval() at test time, so dropout keeps dropping and BatchNorm uses the test batch's statistics.

Self-check: Adam's coupled weight decay against AdamW's

Back to the step rule: weight decay, the previous page's first regularizer, inside Adam (reading 1.6): \(w_a\), \(w_b\) start at 1.0; gradients alternate \(\pm30\), \(\pm0.3\); learning rate 0.01, 500 steps, \(\lambda=0.1\).


after 500 steps (started at 1.0):
  no decay (both):         w_a = 0.9824   w_b = 0.9824
  Adam  (L2-in-gradient):  w_a = 0.9662   w_b = 0.1859
  AdamW (decoupled decay): w_a = 0.5958   w_b = 0.5958
  pure decay, no data updates: 0.6064

Why does Adam(weight_decay=0.1) shrink \(w_b\) to 0.1859 but \(w_a\) only to 0.9662?

Adam adds \(\lambda w\) to the gradient, so \(\sqrt{\widehat V}\approx\lvert g\rvert\) divides the decay: per step about \(\eta\lambda/\lvert g\rvert\) of \(w\), \(3.3\times10^{-5}\) for \(w_a\), \(3.3\times10^{-3}\) for \(w_b\); over 500 steps \(e^{-0.017}=0.98\) against \(e^{-1.67}=0.19\), times the no-decay 0.9824. AdamW: \(0.999^{500}=0.6064\) for both, times 0.9824.

Block 5: model, loss, backward and step all run inside one loop.

Block 5 · S08: a module holds the parameters, a loader the batches

Block 5 is the map's last box: the loop that runs model, loss, backward and step on every minibatch. S08 builds it from two PyTorch objects:


import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
class TinyMLP(nn.Module):
    def __init__(self, d_in, d_hidden, d_out):
        super().__init__()
        self.fc1 = nn.Linear(d_in, d_hidden)
        self.fc2 = nn.Linear(d_hidden, d_out)
    def forward(self, x):
        h = torch.relu(self.fc1(x))
        return self.fc2(h)
model = TinyMLP(3, 16, 2)
X, y = torch.randn(10, 3), torch.tensor([0, 1] * 5)
train_loader = DataLoader(TensorDataset(X, y),
                          batch_size=4, shuffle=True)
  • Each nn.Linear attribute registers its weight and bias; model.parameters() yields the tensors the optimizer receives.
  • nn.Linear(D, H) stores a \((H,D)\) weight: \(16\cdot3+16+2\cdot16+2=98\) parameters here.
  • Define forward(), call model(x).
  • The loader yields xb \((B,3)\), yb \((B,)\): batches of 4, 4, 2, reshuffled each epoch.

The loop, in order: clear, forward, loss, backward, step

With model, train_loader, X and y from the previous page, plus an optimizer and a loss, S08's loop; each inner pass is one parameter update:


opt = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(5):
    model.train()
    for xb, yb in train_loader:
        opt.zero_grad()
        logits = model(xb)
        loss = loss_fn(logits, yb)
        loss.backward()
        opt.step()
  

model.eval()
with torch.no_grad():
    acc = (model(X).argmax(1) == y).float().mean()
  • zero_grad() clears .grad; backward() adds this batch's gradients into it; step() updates every parameter from them.
  • Evaluation, second block (here on the training tensors; S08 evaluates a held-out loader): model.eval() switches dropout and BatchNorm to evaluation behavior; torch.no_grad() stops graph recording. They are independent; use both (S08).

Why clear first? Because backward() adds: the next page shows it.

Gradients accumulate until something clears them

PyTorch adds into .grad across backward calls too, as S04's reuse rule adds over paths (reading 1.5). S08's one-weight model (seed 0, \(x=2\), \(y=1\), mean squared error), backpropagated three times:


import torch
from torch import nn

torch.manual_seed(0)
net = nn.Linear(1, 1, bias=False)          # one scalar weight, as in S08
x, y = torch.tensor([[2.0]]), torch.tensor([[1.0]])
for k in range(3):
    loss = nn.MSELoss()(net(x), y)
    loss.backward()                        # no zero_grad() in between
    print(f"backward {k + 1}: grad = {net.weight.grad.item():+.4f}")
  

backward 1: grad = -4.0599
backward 2: grad = -8.1198
backward 3: grad = -12.1797
  • The same gradient, added three times; S08's trace printed the first, \(-4.0599\). opt.zero_grad() resets .grad to None.

Self-check: delete zero_grad() from the loop

Train the previous page's one-weight model with S08's loop: SGD, learning rate 0.1, from S08's \(w_0=-0.0075\). Its loss \((2w-1)^2\), with gradient \(4(2w-1)\), is 0 at \(w=0.5\).

Delete opt.zero_grad(). (a) Does anything raise an error? (b) Which gradient does each opt.step() use? (c) What happens to \(w\) and to the loss?

(a) No. (b) After \(t\) iterations .grad holds \(g_1+\cdots+g_t\): every step uses the sum of all gradients so far. (c) \(w\) overshoots 0.5 and the loss swings: after steps 1–4, \(w\) is 0.3985, 0.8857, 1.0643, 0.7915; the loss before each step was 1.0302, 0.0412, 0.5950, 1.2739. With zero_grad(), the loss is 0.0001 by step 4. S12 saw the loss swing too; there (Adam, a CNN) it still ended near 0, so the overfit test misses this bug.

Block 6: the same loop, with a model built for images.

Block 6 · S09: a conv's output size counts the windows that fit

Block 6 is the map's first architecture branch: for images, the model keeps the pixel grid and slides one shared kernel over it. One axis: \(I\) inputs, padding \(P\), kernel size \(k\), stride \(S\); window \(m\) covers padded cells \(mS\) to \(mS+k-1\):

\[mS+k\le I+2P\quad\Rightarrow\quad O=\left\lfloor\frac{I+2P-k}{S}\right\rfloor+1\]
S09's cases, \(I=8\), \(k=3\)\(P=0\), \(S=1\)\(P=1\), \(S=1\)\(P=1\), \(S=2\)
\(O\)\(\lfloor5/1\rfloor+1=6\)\(\lfloor7/1\rfloor+1=8\)\(\lfloor7/2\rfloor+1=4\)
  • The \(+1\) counts window \(m=0\); \(\lfloor\cdot\rfloor\) drops a window that does not fully fit (S09).
  • With \(S=1\) and odd \(k\), \(P=(k-1)/2\) keeps the size.
  • Framework "convolution" is cross-correlation, \(Y_{i,j}=\sum_{u,v}X_{i+u,j+v}K_{u,v}\): the kernel is not flipped. A learned kernel just learns flipped weights; it matters when you copy a hand-designed kernel or compare with scipy.signal.convolve2d, which flips (reading 2.1).

A conv layer's parameter count ignores the image size

The output size depends on the image; the parameter count does not. S09's layer:


import torch
from torch import nn

conv = nn.Conv2d(3, 16, kernel_size=3, padding=1)  # C_in=3, C_out=16, 3x3 kernel, P=1
x = torch.randn(8, 3, 32, 32)                      # (N, C_in, H, W)
y = conv(x)                                        # stride 1 by default
print(conv.weight.shape, conv.bias.shape, y.shape)
print(sum(p.numel() for p in conv.parameters()))   # all weights plus all biases
  

torch.Size([16, 3, 3, 3]) torch.Size([16]) torch.Size([8, 16, 32, 32])
448
  • Kernels \((C_{\text{out}},C_{\text{in}},k_h,k_w)\) plus \(C_{\text{out}}\) biases: \(16\cdot3\cdot3\cdot3+16=448\), for any image size. A dense layer from the same \(3\cdot32\cdot32=3072\) inputs to \(16\cdot32\cdot32=16384\) outputs needs \(3072\cdot16384+16384=50{,}348{,}032\) (S09).
  • A \(1\times1\) conv is a per-pixel linear layer across channels: nn.Conv2d(64, 16, 1) has \(64\cdot16+16=1{,}040\) parameters.

Stacked \(3\times3\) layers see \(1+2L\) pixels per axis

One output reads one \(3\times3\) window, but stacking widens its receptive field \(r\). S09's recurrences; \(j\) is the spacing of neighboring activations in input pixels:

\[j_\ell=j_{\ell-1}S_\ell,\qquad r_\ell=r_{\ell-1}+(k_\ell-1)\,j_{\ell-1},\qquad r_0=j_0=1\]
three \(3\times3\) layersconv 1conv 2conv 3
all strides 1: \(r\)357
middle stride 2: \(j\), \(r\)1, 32, 52, 9
  • All strides 1: after \(L\) layers, \(r=1+2L\) per axis (a \(7\times7\) region at \(L=3\)). After a stride-2 layer, each later \(3\times3\) adds \((3-1)\cdot2=4\).
  • VGG (Simonyan and Zisserman, 2014; S10): two \(3\times3\) layers cover one \(5\times5\) region with one more ReLU and, for \(C\) channels in and out, \(18C^2\) weights against \(25C^2\) (\(C=64\): 73,728 against 102,400).

ResNet: a residual block's Jacobian is \(J_F+I\)

Deeper plain stacks fit their own training data worse (S10). ResNet adds each block's input back; \(J_F\) is the branch's Jacobian and \(dY=\partial L/\partial Y\) (S10: for \(F(X)=XW\), \(dY\,J_F=dY\,W^\top\), block 3's rule):

\[Y=F(X)+X\quad\Rightarrow\quad\partial Y/\partial X=J_F+I,\qquad dX=dY\,J_F+dY\]
  • \(dY\) reaches \(X\) unchanged through the identity term, whatever the branch \(F\) computes.
  • Not a guarantee (S10): the branch term \(dY\,J_F\) can cancel it. \(J_F=-0.9I\) gives \(J_F+I=0.1I\); ten such blocks multiply the gradient by \(10^{-10}\).

Self-check. Ten blocks, each with \(J_F=0.1I\). By what factor is the gradient multiplied, without the skip and with it?

Without: \(0.1^{10}=10^{-10}\). With: \((1.1)^{10}\approx2.59\): the identity term keeps each factor near 1 while the branch is small.

Block 7: training these networks with few labels, and spotting a broken run.

Block 7 · S11: with few labels, reuse a pretrained trunk

Block 7 is the map's practice box. S10's networks learned from a million labels; a project has far fewer. S11's recipe: take a pretrained network, drop its head, attach nn.Linear(D, K) for your \(K\) classes.

trainswhenrisk
freeze (linear probe)the head onlyfew labels; target like the sourcefeatures may not fit an unusual domain
fine-tunesome or all of the trunk too, at a \(10\text{–}100\times\) smaller learning rate than from scratchmore labels, or a different domaina large rate destroys pretrained features
  • ResNet-18 (S11): an 11.2M-parameter trunk; a 3-class head trains \(512\cdot3+3=1{,}539\).
  • S11's miniature (reading 2.3, three seeds): a trunk pretrained on 2,000 images of bars, a 2-class head trained on 24 images of "+" and "x"; inverted contrast erases transfer's lead:

matched domain             frozen transfer = 0.8287, scratch = 0.6707
shifted domain (inverted)  frozen transfer = 0.6347, scratch = 0.6627

Default: freeze first; unfreeze once the data can support it (S11).

Augmentation asserts that a transform keeps the label

The other small-data tool: train on \((a(x),y)\), a fresh transform \(a\) each time. That asserts \(a\) does not change the class.

  • S11's task (reading 2.3): class 0 is an "L", class 1 its mirror image, in \(16\times16\) noise; 32 training examples. A shift keeps the class; a horizontal flip turns an "L" into class 1's shape but keeps label 0. Three seeds:

no augmentation                       test acc = 0.7147 (seeds: 0.706, 0.634, 0.804)
random shifts (label-preserving)      test acc = 0.9633 (seeds: 1.000, 0.896, 0.994)
horizontal flips (label-destroying)   test acc = 0.5173 (seeds: 0.510, 0.462, 0.580)
  • Shifts go from 0.7147 to 0.9633, about 25 points. Flips average 0.5173, against majority-class rates (always predicting the more common test class) of 0.514, 0.510 and 0.508: a coin flip (S11).
  • Other label-destroying cases: "b" flips to "d", "6" rotates to "9" (S11).

An augmentation is valid only if it preserves the label for your task.

S12's recipe: overfit one batch, and judge it by the loss

A wrong flip or a transfer to a shifted domain gives a worse number, never an error; so do most bugs (S12). Its recipe, in order: look at the data; dumb baselines; a toy end-to-end run; overfit one batch; scale up, then regularize; tune coarse to fine.

  • Overfit one batch: 8 images with random labels; a healthy network drives the loss to about 0. Reading 2.4, a correct model and one with a softmax before the loss:

--- correct model ---
step 500: loss = 0.0002, batch acc = 1.00
--- buggy (softmax before loss) ---
step 500: loss = 0.7438, batch acc = 1.00
  • Both reach accuracy 1.00; only the loss shows the bug. Softmax outputs in \([0,1]\), read as logits, let the true class lead by at most 1, so with 4 classes \(L\ge\ln(1+3e^{-1})=0.7437\) (S12; \(\ln\) is S03's \(\log\)).
  • It also catches an optimizer built without model.parameters() and labels re-shuffled every step; it misses a missing zero_grad() (S12).

A one-seed difference smaller than the seed spread is not evidence

Once the pipeline passes, is a better number real? S12 reruns one experiment (a small MLP on S10's XOR-quadrant data, 400 Adam steps; it prints test accuracy), data included, under five seeds (reading 2.5):


seed 0, run 1: 0.8895
seed 0, run 2: 0.8895
seeds 0-4: 0.8895, 0.8720, 0.8670, 0.8865, 0.8670
mean = 0.8764, std = 0.0108, spread = 0.0225
  • Same seed twice: identical. Five seeds: \(0.8895-0.8670=0.0225\), a 2.25-point spread from randomness alone.
  • A one-seed win of 2 points can be noise: claims need 3–5 seeds per setup, with the mean and the variation (S12).
  • Two meta-rules govern every step of the recipe (reading 2.4): change one thing at a time; keep the last working configuration.
  • Tune the learning rate first (it usually matters more than any other single knob, reading 2.4); log configuration, seed and library versions.

Self-check: your model cannot overfit eight examples

Step 4 of S12's recipe, overfit one batch, has failed: after many steps on one batch of eight examples, the training loss stays well above 0.

(a) What have you learned? (b) Which bug on the previous pages gives exactly this symptom, and which two more does S12 list? (c) What is not yet worth trying?

(a) Something between the input tensor and the optimizer step is broken: a healthy network memorizes eight examples.

(b) A softmax before the loss: it stalls near 0.74 with 4 classes (block 7's printout). S12 also lists: an optimizer built without model.parameters(), and labels re-shuffled at every step.

(c) Scaling up, regularization, augmentation, more data: they assume this step passed.

Block 8: the same loop and recipe, with a model for sequences.

Block 8 · S13: one recurrent update, applied at every step

Block 8 is the map's second architecture branch. A sequence has no fixed length, so a recurrent neural network (RNN) reads it one step at a time, carrying a hidden state:

\[\mathbf{h}_t=\tanh\left(W_{hh}\,\mathbf{h}_{t-1}+W_{xh}\,\mathbf{x}_t+\mathbf{b}\right)\]
  • The same \(W_{hh}\), \(W_{xh}\), \(\mathbf{b}\) at every step. Backpropagation through time (BPTT) is backprop on the unrolled loop; \(W_{hh}\)'s gradient sums one term per step (S04's reuse rule).
  • Truncated BPTT passes the state on detached (h = h.detach()): no gradient crosses a window boundary, so a dependency at least as long as the window is never learned. Reading 3.1's run (an LSTM cell, this block's last model): the target is the symbol from LAG = 4 steps back, one of 4 symbols, so chance is 0.250; window 4 stays near chance:

  truncation window  4 (window <= LAG): accuracy = 0.320
  truncation window 16 ( window > LAG): accuracy = 0.999

Inside a window, a loss at step \(T\) reaches step \(t\) only through every state between.

The gradient is a product of Jacobians: \(w^{T-1}\) in the scalar case

Block 3's product along a path, one Jacobian per step (S13 calls it S04's series rule):

\[\frac{\partial\mathbf{h}_T}{\partial\mathbf{h}_t}=\prod_{k=t+1}^{T}\frac{\partial\mathbf{h}_k}{\partial\mathbf{h}_{k-1}},\qquad\frac{\partial\mathbf{h}_k}{\partial\mathbf{h}_{k-1}}=\mathrm{diag}\left(1-\mathbf{h}_k^2\right)W_{hh}\]
  • One hidden unit, no tanh: \(h_k=w\,h_{k-1}+x_k\), every factor is \(w\), so \(\partial h_T/\partial h_1=w^{T-1}\). Reading 3.1's rows (S13); at \(T=50\), \(0.9^{49}\) and \(1.1^{49}\):

  w =  0.9: T=5 -> 0.6561   T=20 -> 1.351e-01   T=50 -> 5.726e-03
  w =  1.1: T=5 -> 1.4641   T=20 -> 6.116e+00   T=50 -> 1.067e+02
  • With tanh, each factor is \(w\,(1-h_k^2)\), and \(1-h_k^2\in(0,1]\) since \(\lvert h_k\rvert<1\): no factor exceeds \(\lvert w\rvert\). Saturated, they fall below 1: at \(w=2.0\), 50 steps give \(4.0\times10^{-36}\) (S13).

Linear: \(\lvert w\rvert<1\) vanishes, \(\lvert w\rvert>1\) explodes. With tanh, a saturated \(h\) makes it vanish.

Clipping rescales a gradient whose norm exceeds \(c\)

While \(h\) stays off saturation, \(\lvert w\rvert>1\) can still explode. Of the two failures, exploding has a one-line patch (S13; reading 2.4): after loss.backward(), before opt.step(), torch.nn.utils.clip_grad_norm_(params, c) applies, with \(g\) all parameters' gradients as one vector:

\[g\leftarrow g\cdot\min\left(1,\ c/\lVert g\rVert\right)\]
  • Worked: \(g=(30,40)\), \(\lVert g\rVert=\sqrt{30^2+40^2}=50\). With \(c=1\) the factor is \(\min(1,1/50)=0.02\): \(g\leftarrow(0.6,0.8)\), norm 1, same direction.
  • S13's run (reading 3.1): an RNN reads 60 standard-normal numbers and must output their sum; always predicting 0 scores a mean squared error of about 60, the sum's variance. Unclipped, the loss is nan within 100 steps. Clipped at 1.0, raw norms still reach 7806.4, but the median loss levels off at 33 to 35: the run survives, without learning the long sum.

Clipping bounds the update; what it does to a small gradient is the block's self-check.

LSTM: along the cell, backprop multiplies by a learned gate

Vanishing has no patch inside the vanilla RNN. The long short-term memory (LSTM, S13) adds a cell state updated by gated addition:

\[\mathbf{c}_t=\mathbf{f}\odot\mathbf{c}_{t-1}+\mathbf{i}\odot\mathbf{g},\qquad\mathbf{h}_t=\mathbf{o}\odot\tanh(\mathbf{c}_t)\]
  • Gates \(\mathbf{f},\mathbf{i},\mathbf{o}=\sigma(W[\mathbf{h},\mathbf{x}]+\mathbf{b})\), each with its own \(W\), \(\mathbf{b}\), entries in \((0,1)\) (\([\mathbf{h},\mathbf{x}]\): \(\mathbf{h}_{t-1}\), \(\mathbf{x}_t\) concatenated); candidate \(\mathbf{g}\) via tanh.
  • Gates held fixed, \(\partial\mathbf{c}_t/\partial\mathbf{c}_{t-1}=\mathrm{diag}(\mathbf{f})\): no \(W_{hh}\), no tanh derivative on this direct path.
  • \(\mathbf{f}\) is learned and starts near \(\sigma(\mathbf{b}_f)\): about 0.5 with PyTorch's forget bias near 0, \(\sigma(2)\approx0.88\) with bias 2. S13's gradient norm at the first input (reading 3.1; fb=2: forget bias 2):

   T         RNN        LSTM   LSTM fb=2
  80    5.47e-29    5.27e-18    3.12e-03

Gating hands the decay rate to the model, and to its initialization.

Why the field moved on: every fix keeps the relay

Truncation, clipping and gating all keep one property: information and gradient travel one step at a time.

  • The relay: step 3 to step 300 is 297 multiplications; gating makes it survivable, not free.
  • Serial compute: \(\mathbf{h}_t\) waits for \(\mathbf{h}_{t-1}\): \(T\) steps, one after another.

Self-check. Clipping multiplies every parameter gradient by \(\min(1,c/\lVert g\rVert)\). In \(W_{hh}\)'s gradient, the step-1 term carries the factor \(4.0\times10^{-36}\). Can clipping raise it? Why can clipping bound an exploding update but not raise a vanished one?

No: the factor is at most 1 and multiplies every term alike, so the step-1 term keeps its size, or shrinks further when recent steps push \(\lVert g\rVert\) above \(c\). A vanished gradient needs a path that does not shrink it, such as the LSTM cell with \(\mathbf{f}\) near 1.

Both limits point at one wish: let step 300 look at step 3 directly, and let every step compute at once.

Where this leads: attention replaces the relay

Block 8 ended on the relay and the serial loop. Module 3 continues from there.

  • S14, Attention (reading 3.2): lets one position read any other position directly, instead of through the relay.
  • S16, Transformer I (reading 3.3): self-attention, multi-head attention and positional encoding, with every step computed at once.
  • S13's material is on Quiz 3 (at S23); the Module 1–2 skills carry HW2 (due at S18) and HW3 (out at S18).
  • A block that still does not make sense: office hours (times on Canvas) or email.

The Transformer is a new model for the map's first box; the other four parts are the ones this deck reviewed.

The map, completed: what each part needs, and where it breaks

The title page asked which sessions built each part, which architectures carry them, and where each part breaks:

block · partwhat to rememberbreaks when
1 · model, S01–S02linear layers collapse; put ReLU between theman activation is missing
2 · loss, S03\(-\log p_y\), gradient \(\mathbf{p}-\operatorname{onehot}(y)\); subtract the maxexp overflows
3 · gradient, S04–S05multiply along a path, add over paths; one reverse sweepa reused node's gradient is overwritten
4 · step rule, keep it
working, S06–S07		\(\eta<2/a\); \(v\to g/(1-\beta)\); Adam steps about \(\eta\); AdamW; \(\sigma^2=2/D\) under ReLU\(\eta\) too large; coupled decay; Xavier under ReLU
5 · loop, S08clear, forward, loss, backward, stepzero_grad() or model.eval() is missing
6 · images, S09–S10\(O=\lfloor(I+2P-k)/S\rfloor+1\); \(r=1+2L\); \(\partial Y/\partial X=J_F+I\)a deep plain stack trains worse
7 · practice, S11–S12freeze first; label-preserving augmentation; overfit one batch; 3–5 seedsa flip changes the class; a one-seed claim
8 · sequences, S13\(w^{T-1}\); clipping bounds explosions; cell path \(\mathrm{diag}(\mathbf{f})\)the product vanishes; the window is too short

The review guide's checklist: eight skills, and the block for each

The S15 review guide ends with what you should now be able to do:

after this review you should be able toblock
Show that stacking linear layers without activations stays linear, and read an MLP as hidden units with one hyperplane each plus a linear readout.1
Explain why cross-entropy beats MSE for classification and compute a numerically stable softmax.2
Push gradients through a small computation graph by hand, summing over paths, and say what .grad means.3
Write the momentum and Adam update sketches and distinguish AdamW's weight decay from Adam(weight_decay=...).4
Order the PyTorch training loop from memory and predict what breaks if zero_grad() or model.eval() is missing.5, 4
Compute conv output sizes, parameter counts, and stacked-\(3\times3\) receptive fields; explain what the residual connection fixes.6
Choose freeze vs fine-tune for a given data budget and identify a label-destroying augmentation.7
Explain why RNN long-range gradients vanish or explode, and why the LSTM's forget gate makes decay a decision, not a law.8