Session S04

Backpropagation I

Computation graphs and the chain rule

Deep Learning — CS 4715/5715 · University of Idaho

📦 HW1 out today: micro-autograd

Build a scalar automatic-differentiation engine, the machinery of today's lecture. The whole workflow, top to bottom:

1. Download the starter zip from the Canvas assignment and unzip it
2. Create a venv, pip install -r requirements.txt, and work in hw1_autograd.py
3. Run python -m pytest tests -v until it is green — the same tests grade you
4. Upload hw1_autograd.py to the Canvas assignment HW1 Code; paste WRITEUP.md into HW1 Write-up — resubmit freely, the last submission counts

Upload only that one file: no zip, no tests, no venv. The same steps are in the HW1 README.

HW1: deadline and points

• Out today (S04) · due S08 — then a 12-hour grace window, then the late penalty: −10% per day, floor 60%
• 85 points autograded and 15 points for WRITEUP.md
• No PyTorch needed — the torch parity test showing skipped is expected and costs 0 points
• All tests run on CPU in seconds — the suite is largely gradient checks like today's

HW1: policy, and a sanity check

• Tier 1 AI policy: conceptual/syntax questions OK; no AI-written solution code; include your AI-use statement
• Sanity check from today's lecture: a * a + a at \(a = 3\) must give a.grad == 7
• Reading 1.5 (Inside an Autograd Engine) builds the engine in about 100 lines; the next page says which tasks need it

HW1: start today, finish after session five

• You can start now. Every operation and its local derivative (tasks 1–5) uses only what this session covers, plus += for reused nodes
• One piece waits. backward() (task 6) needs a topological order of the graph: that is in reading 1.5, and session five walks through the engine line by line
• Task 7, fitting a line, is session three's gradient descent, run on your own engine
• That is why the window is long: out today, due at session eight. Read 1.5 this week, watch session five, then finish

Why backpropagation: the cost of a gradient

• S03's promise: give gradient descent \(\nabla_\theta L\) for every parameter, and it trains anything
• The obstacle is scale. With millions of parameters, nudging each one and re-running the network costs one forward pass per parameter
• Backpropagation delivers all the gradients in one backward pass that costs about two forward passes, whatever the parameter count
• That multiple, about two, is fixed. The forward pass itself still grows with the model, so a bigger model still costs more
• One forward pass plus one backward pass gives the gradient of every parameter, so a training step costs about three forward passes, not a million. That is what makes training a large network possible

The chain rule, organized well

• For about seventeen years after the perceptron, hidden layers sat untrainable (S02)
• The credit-assignment gap: when the output is wrong, which hidden weight deserves the blame?
• The answer, when it came, was not new mathematics. It was the chain rule, organized well

"Organized well" is the whole game today: computation graphs, two structural rules, one backward pass.

Computation graphs: every op is a node

• Break any computation into primitive ops (add, multiply, \(\tanh\), matmul). Each intermediate result is a node, and edges run from inputs to outputs
• Any network, together with its loss, becomes a graph whose final node is the scalar loss \(L\)

Differentiation becomes local: the path rule

• The payoff is that differentiation becomes local: each op needs to know only its own derivative with respect to its inputs

The warm-up's three local derivatives

Backprop needs exactly one fact per primitive op: its derivative with respect to each input, evaluated at the stored forward values. Later pages cite these rows by name: the multiply row, the add row, the square row.

• These three rows are the only calculus the warm-up graph needs; the next page multiplies them along paths
• Once the forward pass has run, every entry is a number: \(x = 3,\; w = 2,\; 2b = 14\)

The warm-up graph, drawn: values above, gradients below

What a node stores, and what an op stores

• A node stores \(\partial L/\partial(\text{node})\): the local derivatives multiplied along the path from \(L\) down to it, summed over paths. Every orange number on the last page is one of these
• An op stores its local derivative, \(\partial(\text{output})/\partial(\text{input})\), as a rule it applies when the backward pass reaches it. The derivative between two neighbouring nodes lives on the op, not on a node

The warm-up graph, with numbers

\(a = w x\), \(\;b = a + c\), \(\;L = b^2\), with \(w = 2,\; x = 3,\; c = 1\)

Reusing \(\partial L/\partial b = 14\) for all three inputs is why backprop is cheap: the shared prefix is paid for once.

Forward mode vs. reverse mode: one sweep per input, or one sweep in total

• Forward mode propagates "how does everything change per unit change of this one input" toward the loss: one sweep per input
• Reverse mode propagates "how does the loss change per unit change of this node" backward: one sweep total, giving \(\partial L/\partial(\text{node})\) for every node at once
• A network has millions of inputs (the parameters) and one output (the loss), so reverse mode wins by a factor of millions
• Backprop is reverse-mode automatic differentiation on the network's graph

Who found backpropagation, and when

• Seppo Linnainmaa wrote reverse-mode differentiation down in 1970, in a master's thesis with no neural networks in it
• Paul Werbos applied it to networks in his 1974 PhD thesis
• Both went unnoticed for years
• Rumelhart, Hinton and Williams derived it independently in 1986 and showed it working on real learning problems
• Ideas arrive when someone shows them working, not when they are first written down

Two structural rules: series multiplies, reuse adds

• The quantity \(\partial L/\partial(\text{node})\) is the node's gradient — every framework's .grad is exactly this
• Series (nodes in a chain): a node's gradient is the downstream gradient (the gradient of the node one step closer to \(L\)) times its local derivative
• Reuse (one node feeding several ops; the reading calls this fan-out): the node's gradient is the sum of the contributions coming back along each path
• Everything else is arithmetic

One node used twice: the smallest reuse case

Skip the sum and you silently get \(x\): half the gradient, no error message — the loss still goes down, just slower and wrong.

• This is why autograd engines accumulate with grad +=, never grad =
• HW1's test: a * a + a at \(a = 3\) must give a.grad == 7, which is \(2a + 1\); 4 or 6 means overwriting, not accumulating

Where node reuse returns this semester

• A recurrent network uses the same weight matrix at every timestep: one parameter, many consumers, and the gradients from every timestep add up
• A shared embedding serves both the input layer and the output layer of a language model: two consumers of one table, two contributions to sum
• A skip connection feeds one tensor into both the main branch and the shortcut: two paths back to the loss, one sum
• In every case the rule is the same: gradients from all consumers add. That is the += you write in HW1

The tiny network: four lines, three parameters (reading 1.4)

• The smallest network that exercises every rule: one input, one \(\tanh\) hidden unit, one linear output and a squared loss, with three parameters (subscripts are layer numbers)
• The plan is one sentence: the forward pass computes and stores every intermediate, and the backward pass consumes them in reverse order

The tiny network: the forward pass, with values

# ---- forward pass: compute and STORE every intermediate ----
z = w1 * x
h = np.tanh(z)
yhat = w2 * h + b2
L = 0.5 * (yhat - y) ** 2
      

"Store" is the word that matters: \(h\) and \(\hat{y}\) are exactly what the backward pass will consume. \(z\) is never needed again, because tanh's derivative reads off the stored output, \(1 - h^2\).

The tiny network, drawn: the forward pass on the graph

The backward pass, drawn: seven gradients to fill in

The backward pass begins: four local derivatives

One row per line of the network; the backward pages cite them by name: the loss row, the affine row, the tanh row, the multiply row.

• At the stored values: \(\hat{y} - y = -1.6424\), \(h = 0.7616\), \(w_2 = -1.5\), \(1 - h^2 = 0.4200\), \(x = 2\). Every backward factor is on this line

Backward I — the base case, then the first link

• The half cancels the two; what remains is the error itself, read off the stored \(\hat{y}\)
• Reading 1.5 calls this the base case of the chain rule, and one HW1 test checks only that backward() sets it

Backward II — the affine row: b2 and w2

• \(b_2\)'s local derivative is 1, so the gradient passes through unchanged
• \(w_2\)'s local derivative is the stored \(h = 0.7616\): the first consumption of the forward pass
• Two of three parameter gradients are done; \(w_1\) needs three more links

Backward III — through tanh, to w1

• Every factor is a stored forward value or a parameter
• The intermediates are rounded to four places; multiply the rounded numbers and the last digit drifts (1.0347, 2.0694). The boxed 2.0693 is the unrounded chain

The backward pass, drawn: every gradient filled in

Backward IV — the check

def loss_at(w1_val):
    return 0.5 * (w2 * np.tanh(w1_val * x) + b2 - y) ** 2

eps = 1e-6
fd = (loss_at(w1 + eps) - loss_at(w1 - eps)) / (2 * eps)
      

finite difference for dL/dw1: 2.0693      — same number, two roads
      

• Nudge \(w_1\) by \(10^{-6}\) each way, take the slope: the analytic chain matches to four decimals
• Same number, two roads: a chain of three multiplications, and a plain finite difference that used no calculus at all

Backward V — why the sign is positive

• A gradient says which way \(L\) grows, always: uphill for the loss, never "toward the answer"
• Getting to the answer is gradient descent's job: \(w_1 \leftarrow w_1 - \eta\, \partial L/\partial w_1\) lowers \(w_1\), so \(h\) falls and \(\hat{y}\) rises toward \(y\)
• The sign came from \(w_2 < 0\); with a positive \(w_2\) the same nudge would raise \(\hat{y}\) and the gradient would be negative

Backprop's memory cost is the forward pass

• Every backward factor was either a stored forward value (\(h\), \(\hat{y}\)) or a parameter
• So the backward pass needs every activation the forward pass produced, and a framework must keep all of them alive until the backward pass finishes
• That memory grows with the batch size, the width and the depth of the network
• This is why big models run out of memory in the backward pass, not the forward one

Three ways to differentiate

• Symbolic differentiation writes the derivative out as one formula. It is exact, but the formula grows exponentially with depth unless shared pieces are reused
• Numerical differentiation nudges each parameter and re-runs the network. It costs one forward pass per parameter, the cost we cannot pay
• Backprop is exact like the symbolic answer and costs about as much as a single forward pass. It is the third way, and the only one that scales
• Internalize this before HW1: you are building the third way yourself

Gradient checking: the two formulas

• The central difference has error \(O(\varepsilon^2)\): with \(\varepsilon \sim 10^{-5}\) in float64, expect agreement to about \(10^{-8}\)
• Too small an \(\varepsilon\) and float rounding takes over, which is why \(10^{-5}\) and not smaller
• Absolute error is meaningless across scales; the relative error is the number you read

Why central, and why epsilon squared

• The leading error is \(\tfrac{\varepsilon^2}{6} L'''\), which is the \(O(\varepsilon^2)\): halve \(\varepsilon\) and the error drops by four
• A one-sided difference, \(\big(L(p + \varepsilon) - L(p)\big)/\varepsilon\), keeps the \(\tfrac{\varepsilon}{2} L''\) term and is only \(O(\varepsilon)\)

Gradient checking: four practices

• The single highest-value debugging tool in this course, in four practices:
• Use central differences, not one-sided ones
• Compare with the relative error, not the absolute one
• Check on small inputs. It costs two forward passes per parameter entry, so use toy shapes; correctness transfers. It is a test, not a training algorithm
• Check after every change. HW1 wires this into the tests

Gradient checking: how to read the number

• Below \(10^{-6}\): trust the gradient
• Near \(10^{0}\): it is wrong
• In between: suspect the numerics first (\(\varepsilon\), float32, a ReLU kink at the nudge point), then the math
• You just watched this tool verify a real gradient

For next time

• Read: finish 1.4 Backpropagation (vectorized section) and 1.5 Inside an Autograd Engine — read 1.5 before starting HW1
• No class Monday (Labor Day)
• S05: Backpropagation II — vectorized backprop; how autograd engines work
• HW1 due S08 — start early; the gradient checker is your friend