Session S13

Sequence Modeling: RNNs and LSTMs

A loss at one step of a sequence: how far back can its gradient reach?

Deep Learning · CS 4715/5715 · University of Idaho

Quiz 2 (Module 2, S09–S12) in Canvas at the end of class, 6:45–7:15 PM.

Module 3: a sequence has no fixed length, so read it one step at a time

  • Our MLPs and CNNs mostly took fixed-size inputs. A sentence can be 5 words or 50.
  • A word can depend on one far back: in "The keys to the cabinet are here", are agrees with keys, four words earlier, not with the nearer cabinet.
  • The recurrent neural network (RNN) reads one step at a time and carries a running summary, the hidden state \(\mathbf{h}_t\):
\[\mathbf{h}_t=\tanh\left(W_{hh}\,\mathbf{h}_{t-1}+W_{xh}\,\mathbf{x}_t+\mathbf{b}\right)\]
  • The same \(W_{hh}\), \(W_{xh}\), \(\mathbf{b}\) at every step (weight sharing), as a CNN reuses one kernel at every location.
  • Elman (1990): the hidden states of this network discovered word boundaries and rough syntactic categories.

One network, any length: a 50-word sentence applies the same update 50 times.

What you read off the hidden state, and where its loss sits

\(\mathbf{x}_t\) has \(d_x\) entries and \(\mathbf{h}_t\) has \(d_h\): \(W_{xh}\) is \(d_h\times d_x\), \(W_{hh}\) is \(d_h\times d_h\). A readout \(\hat{\mathbf{y}}_t=W_{hy}\,\mathbf{h}_t+\mathbf{b}_y\), with \(W_{hy}\) of shape \(K\times d_h\) for \(K\) outputs.

read offtaskwhere the loss sits
every \(\mathbf{h}_t\)tagging each worda loss at every step
only \(\mathbf{h}_T\)classifying the whole sentenceone loss, at the last step \(T\)
every \(\mathbf{h}_t\), target \(\mathbf{x}_{t+1}\)a language model: predict the next element (this module's main case)a loss at every step

In code a step takes a \(B\times d_x\) input and a \(B\times d_h\) state, a row per sequence (slide 7: \(B=1\)).

In PyTorch, nn.RNNCell(32, 32) stores \(\mathbf{b}\) as two vectors that add, bias_ih + bias_hh: \(2\cdot32\cdot32+2\cdot32=2{,}112\) parameters.

Today's question: each readout puts a loss on some \(\mathbf{h}_t\). How far back can that loss's gradient reach?

Follow the gradient: unroll the loop, then backpropagate through time

To answer the question, follow the gradient back from a loss. First, how do you train a network that loops?

  • Unroll it: \(T\) steps become a \(T\)-layer feedforward net whose layers all use the same \(W_{hh}\), \(W_{xh}\), each with its own input \(\mathbf{x}_t\). Unrolling adds no parameters.
cell ht xt = h0 Whh, Wxh x1 h1 Whh, Wxh x2 h2 Whh, Wxh x3 h3 L backward: from L, one step at a time, to every earlier state
  • Ordinary backprop on the unrolled graph is backpropagation through time (BPTT), laid out by Werbos (1990).

A loss at step \(T\) reaches step \(t\) only back through every state in between, a step-by-step relay. The next pages show two ways it is cut.

Four ways to wire an unrolled RNN: where inputs enter, where the loss sits

many-to-one h0 x1 x2 x3 L sentiment classification one-to-many h0 = x L L L image captioning (x: a CNN image feature) many-to-many, aligned language model: target xt+1 h0 x1 x2 x3 L L L part-of-speech tagging many-to-many, unaligned h0 x1 x2 x3 L L encoder decoder translation (S14)

The shape of input and output decides where the loss sits; the next pages follow its gradient back.

First cut, a deliberate one: truncated BPTT

The first reason a gradient stops short: the graph is cut on purpose. Nobody unrolls a whole book: memory and compute grow with the number of unrolled steps. With a loss at every step, a long text is read as one stream and cut every 128 steps: backpropagate and update at the end of each window, then pass the state on detached.

  • Unroll a window (say 128 steps), backpropagate inside it, and pass the state on detached (S08): h = h.detach().
  • No gradient crosses a boundary. A target whose source token lies in an earlier window has no gradient path to it, so a dependency at least as long as the window is never learned.
window 4 123456789101112 ✕ cut window 8 123456789101112 path intact

Token 7's target is token 3's symbol, four steps back; dashed lines are window boundaries.

Pick the window longer than the longest dependency the model must learn.

Reading 3.1's run: a 4-step dependency needs a window longer than 4

Excerpt of reading 3.1's train(window): cell, a long short-term memory (LSTM) cell, nn.LSTMCell(4, 64), carrying two states h and c (slide 15), and a linear head read one-hot symbols (V = 4 kinds, chance 0.250); the target is the symbol from LAG = 4 steps back, and history holds the last LAG symbols.


for w in range(n_windows):
    h, c = h.detach(), c.detach()   # carry the state, not the gradient
    loss = 0.0
    for _ in range(window):
        sym = torch.randint(0, V, (1,))
        x = torch.zeros(1, V); x[0, sym] = 1.0
        h, c = cell(x, (h, c))
        logits = head(h)
        target = history[0].view(1)          # the symbol from LAG steps ago
        loss = loss + nn.functional.cross_entropy(logits, target)
        history = torch.cat([history[1:], sym])
    opt.zero_grad(); loss.backward(); opt.step()
  

dependency length LAG = 4, chance accuracy = 0.250
  truncation window  2 (window <= LAG): accuracy = 0.315
  truncation window  4 (window <= LAG): accuracy = 0.320
  truncation window  8 ( window > LAG): accuracy = 0.775
  truncation window 16 ( window > LAG): accuracy = 0.999
  • n_windows = 1,200 windows each, opt = Adam; accuracy over the last 100.
  • Windows 2 and 4: near chance. Window 8: only the later half of each window has a path.

Second cut, an arithmetic one: a product of Jacobians

Inside a window, the gradient still crosses every step: a product. S04's two rules decide what reaches step \(t\).

  • Reuse adds: \(W_{hh}\) is used at every step, so its gradient is a sum of per-step contributions.
  • Series multiplies: a loss at step \(T\) reaches step \(t\) through one Jacobian per hop, a product of \(T-t\) Jacobians:
\[\frac{\partial\mathbf{h}_T}{\partial\mathbf{h}_t}=\prod_{k=t+1}^{T}\frac{\partial\mathbf{h}_k}{\partial\mathbf{h}_{k-1}},\qquad \frac{\partial\mathbf{h}_k}{\partial\mathbf{h}_{k-1}}=\mathrm{diag}\left(1-\mathbf{h}_k^2\right)W_{hh}\]
  • \(1-\mathbf{h}_k^2\) is S04's tanh row, elementwise.
  • Each factor is S10's per-layer factor, but with the same \(W_{hh}\) at every step: the product behaves like a power of one matrix, so it decays or blows up.

A dependency \(T-t\) steps long is learned through a product of \(T-t\) factors.

The scalar case: each factor is \(w\), so the product is \(w^{T-t}\)

Derivation. Prerequisites: S04's series rule and tanh row. One hidden unit, so every Jacobian is a number.

  1. Linear (no activation): \(h_k=w\,h_{k-1}+x_k\) gives \(\partial h_k/\partial h_{k-1}=w\).
  2. Series multiplies: \(\partial h_T/\partial h_t=w\cdot w\cdots w=w^{T-t}\), a product of \(T-t\) factors.
  3. With \(\tanh\): \(h_k=\tanh(w\,h_{k-1}+x_k)\) gives \(\partial h_k/\partial h_{k-1}=w\,(1-h_k^2)\), by the tanh row.
  4. Since \(\lvert h_k\rvert<1\), the tanh derivative \(1-h_k^2\) lies in \((0,1]\), so \(\lvert w\,(1-h_k^2)\rvert\le\lvert w\rvert\): it can only shrink a factor.
  5. Back to vectors: each factor is \(\mathrm{diag}(1-\mathbf{h}_k^2)\,W_{hh}\) (slide 8). Its diagonal entries lie in \((0,1]\), so the tanh part never lengthens a vector, and \(W_{hh}\) takes the role of \(w\).

Linear case: \(\lvert w\rvert<1\) vanishes, \(\lvert w\rvert>1\) explodes. With tanh, no factor exceeds \(\lvert w\rvert\).

The linear case in numbers: \(w^{T-1}\) vanishes below 1 and explodes above it

Reading 3.1's script, Part 1: the scalar case from \(t=1\), so the exponent is \(T-1\).


print("Linear recurrence: dh_T/dh_1 = w^(T-1)")
for w in [0.5, 0.9, 1.0, 1.1, 2.0]:
    print(f"  w = {w:>4}: T=5 -> {w**4:.4f}   T=20 -> {w**19:.3e}   T=50 -> {w**49:.3e}")
  

Linear recurrence: dh_T/dh_1 = w^(T-1)
  w =  0.5: T=5 -> 0.0625   T=20 -> 1.907e-06   T=50 -> 1.776e-15
  w =  0.9: T=5 -> 0.6561   T=20 -> 1.351e-01   T=50 -> 5.726e-03
  w =  1.0: T=5 -> 1.0000   T=20 -> 1.000e+00   T=50 -> 1.000e+00
  w =  1.1: T=5 -> 1.4641   T=20 -> 6.116e+00   T=50 -> 1.067e+02
  w =  2.0: T=5 -> 16.0000   T=20 -> 5.243e+05   T=50 -> 5.629e+14

\(w=0.9\): about 0.006 at \(T=50\), about 100× below the 5-step 0.66. \(w=2.0\): 5.6e14. Only \(w=1.0\) keeps the size; training does not hold a weight there.

Predict for \(h_t=\tanh(w\,h_{t-1}+x_t)\), \(h_0=0\), small random \(x_t\) (std 0.1), \(w=2.0\), \(T=50\): about \(5.6\times10^{14}\), about 1, or near 0?

With tanh, \(w=2.0\) vanishes: once \(h\) saturates, each factor is about 0.16

Answer: near 0. Each factor is \(w\,(1-h_k^2)\) (slide 9, step 3). Follow one run: \(w=2.0\), \(h_0=0\), \(x_k\) with std 0.1.

\(k\)\(x_k\)\(h_k\)\(1-h_k^2\)factor
\(2(1-h_k^2)\)
1+0.1880.1860.9651.93
2−0.1350.2330.9461.89
3−0.1270.3270.8931.79
4+0.0970.6350.5961.19
5−0.1170.8190.3290.66
6+0.1940.9500.0970.19
  • Steps 1–5: \(h\) starts small, factors start near \(w=2\), and the product reaches 5.13.
  • From step 6, \(h\) stays near the fixed point \(h=\tanh(2h)\approx0.96\): factor \(2(1-0.96^2)\approx0.16\) (in this run, 0.10 to 0.23).
  • 45 such factors: \(0.16^{45}\approx1.5\times10^{-36}\); the run's 45 multiply to \(7.8\times10^{-37}\), and \(5.13\times7.8\times10^{-37}=4.0\times10^{-36}\).

Not one factor near 0: 45 factors near 0.16.

Numbers from reading 3.1's script, Part 2, seed 0.

The tanh case in numbers: the product vanishes for all three weights

The same script for three weights (reading 3.1's script, Part 2, seed 0): from \(h_0=0\), the product of the first \(T\) factors \(w\,(1-h_k^2)\).


tanh recurrence: measured |dh_T/dh_0| (product of T per-step factors)
  w =  0.9: T=5 -> 0.3031   T=20 -> 1.066e-02   T=50 -> 1.823e-04
  w =  1.1: T=5 -> 1.3783   T=20 -> 4.979e-02   T=50 -> 1.477e-03
  w =  2.0: T=5 -> 5.1259   T=20 -> 1.806e-12   T=50 -> 4.013e-36
  • Linear: \(\lvert w\rvert<1\) vanishes, \(\lvert w\rvert>1\) explodes. With \(\tanh\): once \(h\) saturates, each factor falls below 1 and the product vanishes (both \(\lvert w\rvert>1\) rows); it explodes when \(h\) stays off saturation (slide 14's run).
  • Bengio, Simard & Frasconi (1994): storing a bit robustly needs dynamics that shrink small perturbations (factors below 1), and that same shrinking makes the gradient vanish. For example, at \(w=2.0\), holding \(h\) near \(\pm0.96\) remembers one yes/no fact; the slope there, about 0.16, pulls noise back and multiplies the gradient at each step by that same 0.16.

Exploding has a one-line patch (next page); vanishing needs the LSTM (slide 15).

The patch for exploding: clip the gradient's norm before each step

Slides 9–12: the product can vanish or explode. Exploding first, since it has a one-line patch; vanishing waits for the LSTM (slide 15). Gradient clipping (Pascanu, Mikolov & Bengio 2013; reading 2.4).

  • After loss.backward() and before opt.step(): if the gradient's norm \(\lVert g\rVert\) exceeds a threshold \(c\), rescale it to norm \(c\):
\[g\leftarrow g\cdot\min\left(1,\ c/\lVert g\rVert\right)\]
  • The direction is kept; \(c\) is a hyperparameter. Reading 3.1's training loop (excerpt):

opt.zero_grad(); loss.backward()
gnorm = torch.nn.utils.clip_grad_norm_(params, clip).item()
max_gnorm = max(max_gnorm, gnorm)
opt.step()
  • clip_grad_norm_ rescales the gradients in place and returns their norm before clipping, the raw norm; clip is 1.0, or float("inf") for no clipping.
  • Called between zero_grad() and backward(), it finds no gradients and returns 0.0.

Clipping keeps a run alive; it does not make it learn the long dependency

Reading 3.1's run: a vanilla RNN reads 60 random numbers and outputs their sum. Carrying the running total undiminished pushes \(W_{hh}\) toward slide 10's \(\lvert w\rvert\ge1\) rows, so the task invites explosion. Always predicting 0 scores a loss of about 60 (a sum of 60 standard normals has variance 60). Same seed, data and learning rate; only the clipping differs.


--- no clipping (largest raw grad norm seen: inf)
  steps   1-100: median loss =          nan
  steps 101-200: median loss =          nan
  steps 201-300: median loss =          nan
  steps 301-400: median loss =          nan
--- clip at 1.0 (largest raw grad norm seen: 7806.4)
  steps   1-100: median loss =      45.3301
  steps 101-200: median loss =      35.3441
  steps 201-300: median loss =      32.7797
  steps 301-400: median loss =      34.6408
  • Unclipped: dead within the first 100 steps.
  • Clipped: raw norms still reach 7806.4, but each update is bounded; the median loss falls below 60 and levels off at 33 to 35.
  • Why it levels off: holding the total for 60 steps is the vanishing side (slide 12), which clipping does not touch: survival, not competence.

Clipping bounds the update; it cannot raise a gradient at the tanh table's \(10^{-36}\). That needs a different architecture.

The fix: an additively updated cell state, run by learned gates (LSTM)

Within the vanilla RNN, vanishing has no patch (slide 12's trade-off); the fix is a different update. Long short-term memory (LSTM): Hochreiter & Schmidhuber 1997. The forget gate \(\mathbf{f}\) was added in 2000 (Gers, Schmidhuber & Cummins); the 1997 paper had none. \([\mathbf{h},\mathbf{x}]\) is the concatenation of \(\mathbf{h}_{t-1}\) and \(\mathbf{x}_t\).

gatecomputed aswhat it does
forget\(\mathbf{f}=\sigma(W_f[\mathbf{h},\mathbf{x}]+\mathbf{b}_f)\)the fraction of each cell entry to keep
input\(\mathbf{i}=\sigma(W_i[\mathbf{h},\mathbf{x}]+\mathbf{b}_i)\)how much of \(\mathbf{g}\) to write
candidate\(\mathbf{g}=\tanh(W_g[\mathbf{h},\mathbf{x}]+\mathbf{b}_g)\)the content that could be written
output\(\mathbf{o}=\sigma(W_o[\mathbf{h},\mathbf{x}]+\mathbf{b}_o)\)how much of the cell to expose
\[\mathbf{c}_t=\mathbf{f}\odot\mathbf{c}_{t-1}+\mathbf{i}\odot\mathbf{g},\qquad \mathbf{h}_t=\mathbf{o}\odot\tanh(\mathbf{c}_t)\]
  • \(\sigma\) outputs lie in \((0,1)\), elementwise: gates scale, they do not switch.
  • Each \(W\) is \(d_h\times(d_h+d_x)\), so \(\mathbf{f},\mathbf{i},\mathbf{g},\mathbf{o},\mathbf{c}_t\) all have \(d_h\) entries.

The cell state is updated by gated addition, not by a matrix product and a tanh.

One LSTM step, forward: keep a fraction, write new content, expose part of it

ct−1 ⊙ + ct σ σ tanh σ f ① forget ⊙ i g ② write tanh ⊙ o ht ③ expose ht−1 xt [h, x] Wf Wi Wg Wo

① forget: keep the fraction \(\mathbf{f}\) of each cell entry. ② write: add \(\mathbf{i}\odot\mathbf{g}\), how much of the candidate \(\mathbf{g}\) to write. ③ expose: \(\mathbf{h}_t=\mathbf{o}\odot\tanh(\mathbf{c}_t)\), how much of the cell to expose.

The gated recurrent unit (GRU, Cho et al. 2014) is a two-gate variant without a separate cell: 3 weight blocks instead of 4; same idea, an additive gated path.

Read the cell left to right: keep a fraction, add new content, expose part of it.

Along the cell, the gradient is multiplied by \(\mathrm{diag}(\mathbf{f})\)

backward along the cell: multiplied by f at the ⊙ node, unchanged at the + node ct−1 ⊙ + ct σ σ tanh σ f ① forget ⊙ i g ② write tanh ⊙ o ht ③ expose ht−1 xt [h, x] Wf Wi Wg Wo
  • With the gates held fixed, \(\partial\mathbf{c}_t/\partial\mathbf{c}_{t-1}=\mathrm{diag}(\mathbf{f})\): no \(W_{hh}\), no \(\tanh\) derivative on this path. The gates depend on \(\mathbf{h}_{t-1}\), so other paths, through the gates, also exist; this is the direct one.
  • S10's residual block had \(\partial Y/\partial X=J_F+I\). The cell path is the same identity path, here scaled by the gate: near \(I\) when \(\mathbf{f}\approx1\).

With \(\mathbf{f}\) near 1, the gradient passes back almost undamped, for as many steps as the gate stays open.

In PyTorch: four gates in one weight matrix, and a forget-gate bias near 0

Slide 17's condition is \(\mathbf{f}\) near 1. At initialization \(W_f[\mathbf{h},\mathbf{x}]\) is small, so \(\mathbf{f}\approx\sigma(\mathbf{b}_f)\): the forget bias sets where \(\mathbf{f}\) starts. Where does PyTorch set it? Input and hidden size 32, as in the measurement that follows; same seed, so the same LSTM initialization.


import torch
from torch import nn
torch.manual_seed(0)
lstm, rnn = nn.LSTMCell(32, 32), nn.RNNCell(32, 32)
for name, p in lstm.named_parameters():
    print(name, tuple(p.shape))
print(sum(p.numel() for p in lstm.parameters()),
      sum(p.numel() for p in rnn.parameters()))
b_f = lstm.bias_ih[32:64] + lstm.bias_hh[32:64]   # the forget slice
print(f"forget-gate bias: {b_f.min():.3f} to {b_f.max():.3f}")
  

weight_ih (128, 32)
weight_hh (128, 32)
bias_ih (128,)
bias_hh (128,)
8448 2112
forget-gate bias: -0.311 to 0.271
  • The four gates' \(W\) stack to \(128\times64\): 8,448 parameters, exactly 4 × the RNN cell's 2,112.
  • Gate order in each 128: input, forget, cell, output. The forget bias starts near 0 (each vector uniform in \(\pm1/\sqrt{32}\approx\pm0.18\)), so \(\mathbf{f}\) starts near 0.5. Raising \(\mathbf{b}_f\) raises \(\mathbf{f}\): \(\sigma(1)\approx0.73\), \(\sigma(2)\approx0.88\).

Measure it: how much gradient reaches the first input?

Does the gradient reach further? Predict, then measure.

Reading 3.1's experiment, seed 0: input and hidden size 32, \(T\) random inputs \(\mathbf{x}_1,\dots,\mathbf{x}_T\), the loss h.sum() on the last hidden state.

  • Three cells: nn.RNNCell(32, 32); nn.LSTMCell(32, 32) as initialized; the same LSTM with forget-gate bias 2 (fb=2).
  • A positive forget bias set by hand is common, usually 1; PyTorch's default sets none (the previous page). The third cell uses 2: 1 in the forget slice of each of bias_ih and bias_hh, so \(\mathbf{f}\) starts near \(\sigma(2)\approx0.88\).
  • Measured: the norm of the gradient at the first input \(\mathbf{x}_1\), for \(T=5,10,20,40,80\).

Predict the order of the three columns at \(T=80\): which cell keeps the most gradient, and which the least? Roughly how far apart?

The LSTM reaches further; forget bias 2 keeps \(3\times10^{-3}\) at \(T=80\)

The printout (reading 3.1): gradient norm at the first input.


   T         RNN        LSTM   LSTM fb=2
   5    9.51e-02    1.13e-01    4.62e-01
  10    2.06e-03    9.79e-03    2.76e-01
  20    2.86e-07    3.54e-05    1.54e-01
  40    4.71e-14    1.64e-09    3.48e-02
  80    5.47e-29    5.27e-18    3.12e-03
  • RNN: about 3.5 orders of magnitude lost per 10 steps; 5e-29 at \(T=80\). No float32 weight moves by that much: the first input is invisible to learning.
  • Default LSTM: four orders of magnitude above the RNN at \(T=40\) in this run, but still geometric: about ×0.6 per step, with untrained forget gates near 0.5.
  • Forget bias 2: \(\mathbf{f}\) starts near 0.88, and a usable 3e-3 reaches the first input at \(T=80\).

Gating hands the decay rate to the model, and to your initialization. In a real run you see these cuts only as symptoms.

In your training run: five symptoms, their causes and fixes

symptomcause from todaycheck or fix
the loss becomes NaNexploding gradientsclip from the start, after backward() and before step(); a smaller learning rate
"Trying to backward through the graph a second time" across batchesthe carried state still holds the previous window's graphh, c = h.detach(), c.detach() at every truncation boundary; not retain_graph=True, which only trades the error for "modified by an inplace operation"
the model uses only the last few tokensvanishing gradients, or a window no longer than the dependencyan LSTM or a gated recurrent unit (GRU), a hand-set forget bias, a longer window
clipping on, loss stable, long-range dependencies not learnedclipping hides vanishinggradient norms at different lags, as S10 printed per-layer norms
sequences padded to one length, the padding processed as real inputthe cell updates its state on pad tokenspack_padded_sequence (gives the cell each sequence's true length), or read the state at each sequence's true last step

Every fix here keeps the relay and the serial loop.

Why the field moved on anyway: the relay and the serial loop

  • Today's answer to slide 3: never past a truncation boundary (slide 6); inside a window, through \(T-t\) factors (\(10^{-36}\) after 50 tanh steps, slide 11); along an LSTM cell, as far as the forget gates stay open (slide 20); always one step at a time.
  • Limitation 1, the relay: token 3 → token 300 is a path of 297 multiplications. Gating makes the relay survivable, not free.
  • Limitation 2, serial compute: \(\mathbf{h}_t\) waits on \(\mathbf{h}_{t-1}\): \(T\) serial steps, and GPUs sit mostly idle.
  • Both point at one wish: let step 300 look directly at step 3, and let every step compute at once.
  • That mechanism is attention: grafted onto RNN translators in 2014–15 (by 2016 Google's translator stacked 8 LSTM layers in encoder and decoder); in 2017 a Google team asked whether the recurrence was needed at all.

Next session (S14): attention.

Self-check: a short memory, a stable loss, an open gate

(a) Your loss is smooth and never NaN, but the LSTM never learns a dependency 20 tokens long. The truncation window is 16. What do you change first?

(b) clip_grad_norm_(params, 1.0) stopped the NaNs. Are the long-range gradients now healthy?

(c) Why does forget bias 2 keep 3e-3 at \(T=80\) when the default LSTM keeps 5e-18?

(a) The window: no gradient crosses a boundary, so no dependency of 16 or more steps is learned (lag 4: window 4 gave 0.320, window 16 gave 0.999, slide 7). Make it longer than the dependency.

(b) Not necessarily: clipping bounds the update; it cannot raise a gradient at 4e-36 (slide 11). Check gradient norms at different lags.

(c) Along the cell, each step multiplies by \(\mathrm{diag}(\mathbf{f})\); bias 2 starts \(\mathbf{f}\) near \(\sigma(2)\approx0.88\) instead of about 0.5 (slides 17 and 20).

For next time

  • Read: 3.2 Attention, from the sequence-to-sequence bottleneck to queries, keys and values.
  • Next session (S14): Attention, the mechanism that deletes the relay.
  • Today's material is on Quiz 3 (S23).

Quiz 2 · the final 30 minutes (6:45–7:15 PM)

Everyone completes Quiz 2 in Canvas.
Covers Module 2: S09–S12.

Online (Web) students and students with flexible-time arrangements may use their assigned flexible completion window.

Closed materials (Tier 0). No late quizzes; make-ups only in the syllabus's three cases.

Before next session (S14): read 3.2 Attention. S14 lets step 300 look directly at step 3; computing every step at once comes with the Transformer (S16).