S15 Review — Guided Review of S01–S13

Module 3 · review guide for self-study · S15 is not held this term

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

Assigned reading: none. This page is a self-study review guide for S01–S13. It was written for S15, the guided-review session, which is not held this term.

How to use this guide

• Work through it in the university’s Midterm Exam Week, the week of S12 and S13: a structured look back over Modules 1–2 and the start of Module 3, half the course, before the Transformer arc begins at S16. Quiz 2 (Module 2) falls in that week, at S13.
• Before Quiz 2 (the last 30 minutes of S13; about 10 questions, 30 minutes, in Canvas; the Online (Web) window is on the Canvas calendar): the S09–S10 and S11–S12 blocks below are its whole scope. Do the S13 block after S13; it is on Quiz 3.
• Midterm grades are reported at S14: your cumulative course grade so far (HW1 and Quizzes 1–2), visible on Canvas before the university’s withdrawal/audit deadline.
• The material reviewed here is still live for later assessment: the skills feed directly into HW2 (due S18) and HW3.

How to review: topic blocks

The blocks below go through S01–S13 in order. For each, one line on what to re-check; follow the link if that line is not already obvious. The one-liners are reminders, not the quiz scope: each reading’s core sections and its Before class you should be able to checklist are what the module quizzes assume.

To test yourself without an answer key, use the decks: the Self-check pages of S09–S12 reveal their answers on click (in S12, released on its class day). Bring anything that still does not make sense to office hours (times on Canvas) or email.

• S01–S02 · from linear models to MLPs — 1.1 Why Deep Learning Works Now, 1.2 From Linear Models to MLPs: re-check why stacked linear layers collapse to one linear map, sigmoid vs ReLU as activations, and dead ReLUs.
• S03 · losses, softmax, learning rates — 1.3 Loss Functions and Gradient Descent: re-check cross-entropy vs MSE for classification, the subtract-the-max softmax stabilization, and the three learning-rate regimes on a loss curve.
• S04–S05 · backpropagation — 1.4 Backpropagation and Computation Graphs, 1.5 Inside an Autograd Engine: re-check local derivatives on a graph, gradients summing over multiple paths, why reverse mode wins, and the finite-difference check from HW1.
• S06–S07 · optimizers, regularization, normalization — 1.6 Optimizers: SGD to AdamW, 1.7 Regularization, Initialization, and Normalization: re-check momentum’s accumulation, the Adam update and the AdamW weight-decay distinction, variance-preserving initialization, and BatchNorm vs LayerNorm axes.
• S08 · PyTorch in practice — 1.8 PyTorch in Practice: re-check the exact order of the training loop, why zero_grad() exists (gradients accumulate), and model.train() vs model.eval() with torch.no_grad().
• S09–S10 · convolution and ResNet — 2.1 Convolution, 2.2 From LeNet to ResNet: re-check the output-size formula and parameter counting, receptive fields of stacked 3x3 convs, the cross-correlation convention, and what the identity path in \(x + F(x)\) does for gradients.
• S11–S12 · transfer learning & training practice — 2.3 Transfer Learning and Data Augmentation, 2.4 The Training Recipe: Debugging Deep Networks, 2.5 Experiments and Reproducibility: re-check freeze vs fine-tune (and the 10–100x smaller learning rate), label-preserving augmentation, the overfit-one-batch keystone test, and why a one-seed difference smaller than the seed-to-seed spread (2.25 points over five seeds in 2.5) is not evidence: claims need 3–5 seeds per setup.
• S13 · RNNs and LSTMs — 3.1 RNNs and LSTMs: re-check the recurrence and BPTT, the powers-of-w vanishing/exploding argument, clipping fixing exploding only, and the LSTM’s additive cell update.

After this review you should be able to

• Show that stacking linear layers without activations stays linear, and read an MLP as hyperplane-carving hidden units plus a linear readout.
• Explain why cross-entropy beats MSE for classification and compute a numerically stable softmax.
• Push gradients through a small computation graph by hand, summing over paths, and say what .grad means.
• Write the momentum and Adam update sketches and distinguish AdamW’s weight decay from Adam(weight_decay=...).
• Order the PyTorch training loop from memory and predict what breaks if zero_grad() or model.eval() is missing.
• Compute conv output sizes, parameter counts, and stacked-3x3 receptive fields; explain what the residual connection fixes.
• Choose freeze vs fine-tune for a given data budget and identify a label-destroying augmentation.
• Explain why RNN long-range gradients vanish or explode, and why the LSTM’s forget gate makes decay a decision, not a law.

Connects to

• The guide covers S01–S13. Quiz 2, at S13, covers Module 2 (S09–S12); the Module 3 material (S13) is part of Quiz 3, given at S23.
• Next reading: 3.3 The Transformer, for S16; Transformer II follows at S17.
• The project pre-proposal is due at the end of the S13 week; HW2 is due at S18, when HW3 goes out.

---

← S14 · Attention · All readings · S16 · Transformer I →