Module 3 · Reading 3.1 · read before S13: RNNs and LSTMs

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

The MLP and CNN examples so far used fixed tensor shapes and mostly fixed-size predictions, even though CNNs can be made spatially flexible with pooling and dense models can process padded or batched variable-length data. Sequence models make variable length and ordered dependence central: a sentence can be five words or fifty, and later meanings depend on earlier context. For roughly three decades, the standard answer was the recurrent neural network (RNN).

RNNs are no longer how the field models language; the Transformer, which you will meet in the S16 reading, replaced them. But you should still understand them, for two reasons. First, they introduce the core vocabulary of sequence modeling (hidden state, unrolling, backpropagation through time) that every later idea assumes. Second, their central failure, the vanishing gradient, is the single best-documented motivation for both the LSTM and, ultimately, attention. This section is the historical bridge: what recurrence got right, exactly where it broke, and why the fix that saved it in 1997 was not enough by 2017.

How to read this

Sequences need memory

A recurrent network processes a sequence \(x_1, x_2, \ldots, x_T\) one element at a time. At each step it combines the new input with a summary of everything seen so far:

\[h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b)\]

The vector \(h_t\) is the hidden state, the network’s memory. The same weight matrices \(W_{hh}\) and \(W_{xh}\) are used at every step, just as a CNN reuses the same kernel at every image location. This weight sharing is what lets one network handle sequences of any length: a 5-word sentence applies the update 5 times, a 50-word sentence applies it 50 times.

From the hidden state you can read off whatever the task needs: a prediction at every step (tagging each word), a prediction only at the end (classifying the whole sentence), or (the case that will matter most in this chapter) a prediction of the next element, which turns the network into a language model.

It helps to name the input/output shapes explicitly, because they organize everything recurrent networks were used for. Many-to-one: read a sequence, emit one label (sentiment classification, spam detection); run the recurrence, keep the last state, attach a classifier. One-to-many: one input, a sequence out (image captioning: a CNN feature vector seeds the initial state, then the RNN generates words). Many-to-many, aligned: one output per input (part-of-speech tagging); predict from every \(h_t\). Many-to-many, unaligned: input and output sequences of different lengths with no per-step correspondence (translation, speech-to-text) — the hardest case, and the one that drove most of the architectural history in this chapter. Language modeling is the aligned many-to-many case with a twist: the target at step \(t\) is the input at step \(t+1\), so the training data labels itself.

Two structural details round out the picture. First, real inputs are rarely raw vectors: for text, \(x_t\) is a learned embedding (a row of a trainable matrix indexed by the token’s id), so the network learns its own input representation jointly with the recurrence (embeddings return in force when this chapter reaches tokenization). Second, RNN layers stack: the sequence of hidden states from one layer becomes the input sequence to the next, giving depth per timestep on top of depth across time. Stacks of two to eight recurrent layers were standard in the systems this section ends with.

The design has deep roots. Jeffrey Elman’s 1990 paper “Finding Structure in Time” popularized this “simple recurrent network” in cognitive science, showing that a network fed one letter or word at a time could discover word boundaries and rough syntactic categories in its hidden state. The architecture you just saw is still often called an Elman network.

Backpropagation through time

How do you train a network that loops? You unroll it. An RNN run for \(T\) steps is exactly equivalent to a \(T\)-layer feedforward network in which every layer has the same weights and each layer also receives an external input \(x_t\). Unrolled, it is an ordinary computation graph, and the ordinary backpropagation you implemented in Chapter 1 applies. Applied to an unrolled recurrence, it is called backpropagation through time (BPTT); Paul Werbos’s 1990 paper of that name laid out the procedure explicitly.

One consequence deserves emphasis. Because \(W_{hh}\) is shared across all \(T\) steps, its gradient is a sum of contributions, one per step. And the contribution that flows from a loss at step \(T\) back to an input at step \(t\) must pass through every intermediate hidden state. Each hop multiplies the gradient by the Jacobian \(dh_k/dh_{k-1}\), so the long-range contribution contains a product of \(T - t\) Jacobians. Each of those Jacobians is \(\mathrm{diag}(1-h_k^2)\,W_{hh}\), and products of many such matrices behave like powers of a matrix: powers either decay or blow up.

In practice, nobody unrolls over an entire book. Truncated BPTT unrolls a window of, say, 128 steps, backpropagates within the window, and carries the hidden state (but not gradients) across windows. This caps memory and compute, at the cost of never training on dependencies longer than the window.

Truncated BPTT, demonstrated

Truncated BPTT cuts direct gradient credit across window boundaries, a limitation that deserves a demonstration because it is invisible in a loss curve. The forward state can still carry information farther than the window, and learned recurrent dynamics may sometimes exploit it indirectly; what is missing is an unbroken backward path assigning a distant input credit for a later prediction. The experiment below constructs a task whose dependency length is known exactly.

We train the same LSTM with different truncation windows, always carrying the hidden state across window boundaries (detached, so no gradient crosses). Think about where the gradient can flow. A prediction made at position j inside a window depends on the input at position j - LAG. If that input fell in a previous window, the path between them was severed by the detach, and backpropagation cannot credit the input for the prediction. Such a path exists for at least one position only when the window is strictly longer than LAG — a window of exactly LAG steps still leaves every target’s source token stranded on the far side of a boundary.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Truncated BPTT on a task with a known dependency length. The input is a
# stream of one-hot symbols; the target at step t is the symbol seen at
# step t - LAG, so the model must hold each symbol for LAG steps. We train
# with several truncation windows, always carrying the hidden state
# (detached) across window boundaries. A gradient path from a prediction
# back to the symbol it depends on exists only when both fall inside the
# same window -- which requires window > LAG.

V, LAG, HIDDEN = 4, 4, 64

def train(window, n_windows=1200):
    torch.manual_seed(0)
    cell = nn.LSTMCell(V, HIDDEN)
    head = nn.Linear(HIDDEN, V)
    opt = torch.optim.Adam(list(cell.parameters()) + list(head.parameters()), lr=1e-2)
    h = torch.zeros(1, HIDDEN); c = torch.zeros(1, HIDDEN)
    history = torch.zeros(LAG, dtype=torch.long)  # the last LAG symbols seen
    correct = total = 0
    for w in range(n_windows):
        h, c = h.detach(), c.detach()   # carry the state, not the gradient
        loss = 0.0
        for _ in range(window):
            sym = torch.randint(0, V, (1,))
            x = torch.zeros(1, V); x[0, sym] = 1.0
            h, c = cell(x, (h, c))
            logits = head(h)
            target = history[0].view(1)          # the symbol from LAG steps ago
            loss = loss + nn.functional.cross_entropy(logits, target)
            if w >= n_windows - 100:             # accuracy over the last 100 windows
                correct += int(logits.argmax(dim=-1).item() == target.item())
                total += 1
            history = torch.cat([history[1:], sym])
        opt.zero_grad(); loss.backward(); opt.step()
    return correct / total

print(f"dependency length LAG = {LAG}, chance accuracy = {1/V:.3f}")
for window in [2, 4, 8, 16]:
    acc = train(window)
    tag = "window > LAG" if window > LAG else "window <= LAG"
    print(f"  truncation window {window:>2} ({tag:>13}): accuracy = {acc:.3f}")

Running this prints:

dependency length LAG = 4, chance accuracy = 0.250
  truncation window  2 (window <= LAG): accuracy = 0.315
  truncation window  4 (window <= LAG): accuracy = 0.320
  truncation window  8 ( window > LAG): accuracy = 0.775
  truncation window 16 ( window > LAG): accuracy = 0.999

The boundary is sharp. With windows of 2 and 4 (at or below the lag), accuracy sits near chance after 1,200 updates, and no amount of further training will fix it: the learning signal that says “store this symbol, you will need it in four steps” does not exist in any gradient the model receives. With a window of 16, the same model, same data, same optimizer reaches 99.9%. The window of 8 lands in between (77.5%): only the later positions in each window have an intact gradient path, so the training signal is diluted but present.

One more detail rewards attention: the short-window runs sit slightly above chance (0.315 versus 0.250). Truncation cuts gradients, not information. The forward pass still carries the past through the hidden state, and the recurrent dynamics happen to preserve a decodable trace of recent symbols even without being trained to; the output head learns to read what is accidentally there. This “reservoir” effect (random recurrent networks retain short-term information for free) was studied in its own right as echo-state networks in the early 2000s. The practical rule stands, though: pick the truncation window longer than the longest dependency you need the model to learn, not merely to use.

Vanishing and exploding gradients, with real numbers

Make the problem concrete with a scalar recurrence, where the Jacobian product is just a product of numbers. For the linear map \(h_t = w\, h_{t-1} + x_t\), the sensitivity of the last state to the first is exactly \(w^{T-1}\). For the realistic tanh version, each backprop step multiplies by \(w\,(1 - h_t^2)\), whose magnitude is at most \(\lvert w\rvert\): the tanh derivative can only shrink the gradient further.

import numpy as np

np.random.seed(0)

# Part 1: linear recurrence h[t] = w * h[t-1] + x[t].
# The gradient dh_T/dh_1 is exactly w^(T-1): pure geometric growth or decay.
print("Linear recurrence: dh_T/dh_1 = w^(T-1)")
for w in [0.5, 0.9, 1.0, 1.1, 2.0]:
    print(f"  w = {w:>4}: T=5 -> {w**4:.4f}   T=20 -> {w**19:.3e}   T=50 -> {w**49:.3e}")

# Part 2: the real thing, h[t] = tanh(w * h[t-1] + x[t]), from h[0] = 0.
# Each backprop step multiplies by  w * tanh'(pre) = w * (1 - h[t]^2), at most |w| in size,
# so the product of the first T factors is dh_T/dh_0.
print("\ntanh recurrence: measured |dh_T/dh_0| (product of T per-step factors)")
for w in [0.9, 1.1, 2.0]:
    h = 0.0
    xs = np.random.randn(50) * 0.1
    factors = []
    for t in range(50):
        h = np.tanh(w * h + xs[t])
        factors.append(w * (1.0 - h ** 2))
    prods = np.cumprod(factors)
    print(f"  w = {w:>4}: T=5 -> {abs(prods[4]):.4f}   "
          f"T=20 -> {abs(prods[19]):.3e}   T=50 -> {abs(prods[49]):.3e}")

Running this prints:

Linear recurrence: dh_T/dh_1 = w^(T-1)
  w =  0.5: T=5 -> 0.0625   T=20 -> 1.907e-06   T=50 -> 1.776e-15
  w =  0.9: T=5 -> 0.6561   T=20 -> 1.351e-01   T=50 -> 5.726e-03
  w =  1.0: T=5 -> 1.0000   T=20 -> 1.000e+00   T=50 -> 1.000e+00
  w =  1.1: T=5 -> 1.4641   T=20 -> 6.116e+00   T=50 -> 1.067e+02
  w =  2.0: T=5 -> 16.0000   T=20 -> 5.243e+05   T=50 -> 5.629e+14

tanh recurrence: measured |dh_T/dh_0| (product of T per-step factors)
  w =  0.9: T=5 -> 0.3031   T=20 -> 1.066e-02   T=50 -> 1.823e-04
  w =  1.1: T=5 -> 1.3783   T=20 -> 4.979e-02   T=50 -> 1.477e-03
  w =  2.0: T=5 -> 5.1259   T=20 -> 1.806e-12   T=50 -> 4.013e-36

Read the linear table first. At w = 0.9 the 50-step gradient is about 0.006: long-range influence has faded a hundredfold relative to short-range. At w = 2.0 it is about 5.6e14: a single step of gradient descent with that gradient destroys the model. Only the knife-edge w = 1.0 preserves gradient magnitude, and training will not keep a weight exactly there.

The tanh rows are worse in an instructive way. Even w = 2.0, which “should” explode, collapses to 4e-36 by step 50: a large recurrent weight drives the hidden state toward saturation (h near ±1), where the tanh derivative is nearly zero. This is the general picture that emerged from analysis in the early 1990s: with saturating nonlinearities, gradients through a recurrence typically vanish, and occasionally explode when the state passes through the unsaturated region. Either way, the network cannot reliably learn that a word 50 steps ago determines the correct output now — the learning signal for that dependency is numerically gone.

Historical note. The problem was diagnosed almost as soon as people tried to train RNNs seriously. Sepp Hochreiter analyzed the vanishing gradient in his 1991 diploma thesis at TU Munich (written in German, so it circulated slowly), and Bengio, Simard, and Frasconi published an independent analysis, “Learning long-term dependencies with gradient descent is difficult,” in 1994. That paper’s framing (the conditions that make an RNN store information reliably are the same conditions that make its gradients vanish) is still the cleanest statement of the dilemma.

For the exploding side there is a workable patch: gradient clipping, analyzed by Pascanu, Mikolov, and Bengio in 2013, rescales the gradient whenever its norm exceeds a threshold. It is a common and cheap safeguard for recurrent training, especially when diagnostics show spikes, but the threshold is a hyperparameter and clipping is not mandatory for every stable recipe. It cannot resurrect gradients that have already vanished.

Gradient clipping in action

Before moving to that architectural fix, watch clipping earn its keep on a live training run. The task below is chosen to invite explosion: a vanilla RNN reads 60 random numbers and must output their sum. Accumulating a sum means the hidden state must carry its running total undiminished across all 60 steps, which pushes the recurrent weight matrix toward the unstable regime: exactly the \(w \ge 1\) territory from the scalar experiment above. The two runs are identical in every respect (same seed, same data order, same learning rate); the only difference is whether the gradient norm is clipped at 1.0 before each update. Since the targets are sums of 60 independent standard normals, a model that learned nothing and always predicted 0 would score a mean squared error near 60 — that is the “knows nothing” baseline to judge the numbers against.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Exploding gradients in practice, and what clipping does. A vanilla RNN
# reads 60 random numbers and must output their running total -- a task
# that pushes the recurrent weights toward instability, because holding a
# sum means the state must not decay. Same data, same initialization, same
# learning rate; the only difference between the two runs is gradient-norm
# clipping. Predicting 0 always would give a loss near 60 (the variance of
# a sum of 60 standard normals), so 60 is the "learned nothing" baseline.

T, B, H = 60, 16, 32

def run(clip):
    torch.manual_seed(0)
    rnn = nn.RNN(1, H, batch_first=True)
    head = nn.Linear(H, 1)
    params = list(rnn.parameters()) + list(head.parameters())
    opt = torch.optim.SGD(params, lr=0.1)
    max_gnorm, losses = 0.0, []
    for step in range(1, 401):
        x = torch.randn(B, T, 1)
        target = x.sum(dim=1)
        out, _ = rnn(x)
        pred = head(out[:, -1, :])
        loss = ((pred - target) ** 2).mean()
        opt.zero_grad(); loss.backward()
        gnorm = torch.nn.utils.clip_grad_norm_(params, clip).item()
        max_gnorm = max(max_gnorm, gnorm)
        opt.step()
        losses.append(loss.item())
    return losses, max_gnorm

for clip, name in [(float("inf"), "no clipping"), (1.0, "clip at 1.0")]:
    losses, max_gnorm = run(clip)
    print(f"--- {name} (largest raw grad norm seen: {max_gnorm:.1f})")
    t = torch.tensor(losses)
    for lo in range(0, 400, 100):
        print(f"  steps {lo + 1:>3}-{lo + 100:>3}: median loss = {t[lo:lo + 100].median().item():>12.4f}")

Running this prints:

--- no clipping (largest raw grad norm seen: inf)
  steps   1-100: median loss =          nan
  steps 101-200: median loss =          nan
  steps 201-300: median loss =          nan
  steps 301-400: median loss =          nan
--- clip at 1.0 (largest raw grad norm seen: 7806.4)
  steps   1-100: median loss =      45.3301
  steps 101-200: median loss =      35.3441
  steps 201-300: median loss =      32.7797
  steps 301-400: median loss =      34.6408

Without clipping, the run is dead within the first hundred steps: one batch produced a gradient large enough that the weight update destroyed the model, the next loss was astronomically large, its gradient larger still, and the process cascaded to inf and then NaN. Nothing in the loss values before the collapse warned of it — this is why exploding gradients feel like a lightning strike when you first meet them. With clipping, the same trajectory survives every spike: the raw gradient norm still reached the thousands (clipping does not prevent large gradients; it prevents large updates), but each update moved the weights by a bounded amount, and the median loss descends well below the baseline of 60.

Look honestly at where the clipped run ends, though: a median loss around 33 after 400 steps is stable, improving — and far from solved. Clipping bought survival, not competence. Holding a value in memory for 60 steps is still fighting the vanishing side of the problem, which no amount of clipping addresses. That is the cue for the architectural fix.

The LSTM: gating what to remember

That fix is the Long Short-Term Memory (LSTM), published by Sepp Hochreiter and Jürgen Schmidhuber in Neural Computation in 1997. The core idea: stop forcing information to survive repeated multiplication by a weight matrix and a squashing nonlinearity. Instead, give the network a separate memory pathway (the cell state \(c_t\)) that is updated additively, and let learned gates decide what enters, what leaves, and what is erased.

The modern LSTM cell computes, at each step (with \([h, x]\) denoting concatenation of \(h_{t-1}\) and \(x_t\), and \(\operatorname{sigmoid}\) producing values in \((0, 1)\)):

\(f = \operatorname{sigmoid}(W_f [h, x] + b_f)\) (forget gate: what fraction of each memory slot to keep)

\(i = \operatorname{sigmoid}(W_i [h, x] + b_i)\) (input gate: how much new content to write)

\(g = \tanh(W_g [h, x] + b_g)\) (candidate content to write)

\(c_t = f \odot c_{t-1} + i \odot g\) (the cell update, elementwise)

\(o = \operatorname{sigmoid}(W_o [h, x] + b_o)\) (output gate)

\(h_t = o \odot \tanh(c_t)\) (the exposed hidden state)

The line that matters is the cell update. Backpropagating through \(c_t = f \odot c_{t-1} + i \odot g\) multiplies the gradient by \(f\): a value the network chooses, per timestep and per memory dimension. When the network sets a forget gate near 1, that memory slot passes gradients back almost undamped, for as many steps as the gate stays open. The recurrence no longer forces geometric decay; it makes decay a learned decision. (One honest historical wrinkle: the 1997 paper had no forget gate — the cell state was never erased. The forget gate was added by Gers, Schmidhuber, and Cummins in 2000, and every LSTM you will use today includes it.)

People. Hochreiter was a student of Schmidhuber’s at TU Munich when he wrote the 1991 thesis identifying the vanishing gradient; the LSTM was their answer to their own diagnosis. The paper drew modest attention for years (sequence tasks that needed long memory were niche in the late 1990s), but by the mid-2010s LSTMs were inside production speech recognition and translation systems at Google and elsewhere, making the 1997 paper one of the most cited in the field’s history.

Measuring the difference

Claims about gradient flow should be measurable, so measure them. The script below runs a vanilla RNN cell and an LSTM cell over sequences of increasing length T, puts a loss on the final hidden state, and asks: how much gradient reaches the first input token? A third column applies the classic practical trick of initializing the forget-gate bias to a positive value, so the forget gate starts mostly open (recommended, among others, by Jozefowicz, Zaremba, and Sutskever’s 2015 empirical study); here the bias is 2 (1 in each of PyTorch’s bias_ih and bias_hh).

import torch
import torch.nn as nn

torch.manual_seed(0)

# How much gradient reaches the FIRST input token from a loss on the LAST
# hidden state, as the sequence gets longer? Vanilla RNN vs LSTM vs LSTM
# with forget-gate bias 2 (1 in each of bias_ih and bias_hh).

d = 32  # hidden size and input size

def grad_to_first_input(cell_type, T):
    torch.manual_seed(0)
    if cell_type == "rnn":
        cell = nn.RNNCell(d, d)  # tanh nonlinearity
    else:
        cell = nn.LSTMCell(d, d)
        if cell_type == "lstm_fb2":
            with torch.no_grad():
                # gate order in the bias vectors is [input, forget, cell, output]
                cell.bias_ih[d:2 * d].fill_(1.0)
                cell.bias_hh[d:2 * d].fill_(1.0)
    xs = [torch.randn(1, d, requires_grad=True) for _ in range(T)]
    h = torch.zeros(1, d)
    c = torch.zeros(1, d)
    for x in xs:
        if cell_type == "rnn":
            h = cell(x, h)
        else:
            h, c = cell(x, (h, c))
    h.sum().backward()
    return xs[0].grad.double().norm().item()

print(f"{'T':>4}  {'RNN':>10}  {'LSTM':>10}  {'LSTM fb=2':>10}")
for T in [5, 10, 20, 40, 80]:
    row = [grad_to_first_input(k, T) for k in ["rnn", "lstm", "lstm_fb2"]]
    print(f"{T:>4}  {row[0]:>10.2e}  {row[1]:>10.2e}  {row[2]:>10.2e}")

Running this prints:

   T         RNN        LSTM   LSTM fb=2
   5    9.51e-02    1.13e-01    4.62e-01
  10    2.06e-03    9.79e-03    2.76e-01
  20    2.86e-07    3.54e-05    1.54e-01
  40    4.71e-14    1.64e-09    3.48e-02
  80    5.47e-29    5.27e-18    3.12e-03

Three observations. First, the vanilla RNN’s gradient decays by roughly 3.5 orders of magnitude every 10 steps, and at T = 80 it is 5e-29: no float32 weight moves by that much, so the first token is invisible to learning. Second, the LSTM at default initialization is better at every length in this run, by over four orders of magnitude at T = 40, but it still decays geometrically, by a factor of about 0.6 per step, while untrained forget gates average about 0.5. Third, opening the forget gates (bias = 2) keeps the gradient at a usable 3e-3 even at 80 steps. Gating does not repeal the mathematics; it hands the decay rate to the model, and to your initialization.

Pitfall. “I added gradient clipping, so my gradient problems are solved.” Clipping addresses exploding gradients only. A clipped training run can look perfectly stable (smooth loss, no NaNs) while long-range gradients have vanished and the model quietly learns only short-range patterns. If a sequence model plateaus, check gradient norms at different lags (as above) rather than assuming stability means health.

Beyond left to right: bidirectional RNNs and CTC

Two extensions of the basic recurrence mattered enough historically (and connect forward to enough of this chapter) that you should know them at least at the level of the idea.

Bidirectional RNNs. An RNN as defined so far is causal: \(h_t\) summarizes the past only. Plenty of tasks have no such constraint. When tagging the word “bank” in “the bank of the river,” the disambiguating evidence comes after the word; a speech recognizer transcribing a syllable benefits from hearing how the word ends. Mike Schuster and Kuldip Paliwal’s 1997 solution runs two independent recurrences over the same input (one left-to-right producing \(h_t^{\text{fwd}}\), one right-to-left producing \(h_t^{\text{bwd}}\)) and concatenates them, so the representation of position \(t\) summarizes the entire sequence with \(t\) as its pivot. In PyTorch this is one flag (bidirectional=True, which doubles the output width). The catch is as important as the trick: a bidirectional network needs the whole input before it can produce any output, so it cannot generate text autoregressively and cannot transcribe speech as it arrives. This exact tension (bidirectional context gives better representations, causality enables generation) reappears at Transformer scale as the BERT-versus-GPT split in section 3.4, and one of the last major RNN-era NLP systems, ELMo (Peters et al., NAACL 2018), was built precisely on forward and backward LSTM language models trained side by side.

CTC. Speech-to-text has a structural problem that word tagging does not: the input (say, 300 audio frames) and the output (say, 20 characters) have different lengths and no known alignment — nobody tells you which frames correspond to which character. For years this forced hybrid systems that bolted neural acoustic models onto alignment machinery. Connectionist Temporal Classification, introduced by Alex Graves and colleagues at ICML 2006, made the alignment a latent variable handled inside the loss. The network emits, at every input frame, a distribution over characters plus a special blank symbol; any frame-level string collapses to an output by removing repeats and then blanks (so hh-e-ll-llo- collapses to hello); and the CTC loss sums the probability of every frame-level string that collapses to the correct transcript, a sum over exponentially many alignments, computed exactly in polynomial time by dynamic programming, much like the forward algorithm for hidden Markov models. The network learns when to commit to a character and when to emit blank, and alignment emerges without ever being labeled. CTC powered the first strong end-to-end neural speech recognizers and remains in production use; you will meet its dynamic-programming style again in section 3.4’s unigram tokenizer.

People. Alex Graves, a student of Schmidhuber’s at IDSIA in Switzerland, is the person most responsible for demonstrating what LSTMs could actually do: CTC (2006) made them transcribe speech and handwriting without alignments, his 2013 work generated strikingly realistic handwriting one pen-coordinate at a time, and his 2012 sequence-transduction paper (RNN-T) became the backbone of streaming speech recognition on phones. Much of the mid-2010s conviction that recurrent networks were the future of sequence modeling traces to this body of work.

Why the field moved on

Through the mid-2010s, LSTMs (and the simplified gated variant, the gated recurrent unit (GRU), introduced by Cho et al. in 2014) were the state of the art for language. The 2014 sequence-to-sequence papers by Sutskever, Vinyals, and Le and by Cho et al. made LSTM encoder–decoders the standard for machine translation, and Google’s 2016 production translation system (Google Neural Machine Translation, GNMT; Wu et al.) stacked eight LSTM layers in both its encoder and decoder.

Yet two limitations proved decisive. The first you have now measured: even with gates, information from far back must survive a step-by-step relay through every intermediate state. The path length between token 3 and token 300 is 297 multiplications; gating makes the relay survivable, not free. The second is computational: \(h_t\) cannot be computed before \(h_{t-1}\), so an RNN processes a sequence in \(T\) inherently serial steps. GPUs earn their keep by doing massive parallel work, and recurrence leaves them mostly idle. As datasets and models grew, training time became dominated by this serial bottleneck.

Both limitations point at the same wish: let step 300 look directly at step 3, without a relay, and let every step’s computation happen at once. That mechanism, attention, was first grafted onto RNN encoder–decoders in 2014–2015, and then in 2017 a Google team asked whether the recurrence was needed at all. The answer, and the architecture built from it, is where this chapter goes next.

Moved on, however, does not mean extinct, and it is worth being precise about where recurrence still earns its place. The properties that made RNNs slow to train make them attractive to run. An RNN processes a stream with constant work and constant memory per step: consume the new input, update the fixed-size state, discard the input. A Transformer generating or transcribing with full attention must keep the representations of everything it has seen (its memory footprint grows with the sequence), and each new step’s attention touches all of it. For always-listening, low-latency, on-device workloads, the recurrent trade is often the right one, and it is no accident that streaming speech recognition held out longest: the recognizer Google shipped on the Pixel 4 in 2019 for on-device dictation was built on the RNN transducer (RNN-T), a Graves-designed recurrent architecture that emits text incrementally as audio arrives. Recurrence also persists where sequences are extremely long relative to the information they carry (sensor streams, some time-series forecasting), where the quadratic cost you will measure in the next section bites hardest.

There is also a quieter reason to learn this material: the concepts outlived the architecture. Hidden state as a running summary, unrolling in time, detaching to truncate a gradient, gating as learned information flow — each reappears in modern systems under new names, from the KV-cache of a generating Transformer (a kind of growing state) to the gates of modern state-space models. The vocabulary you built in this section is not historical trivia; it is the working language of sequence modeling.

Going deeper

The dilemma is fundamental, not incidental. The 1994 Bengio–Simard–Frasconi result mentioned above is sharper than “gradients tend to shrink.” It argues that for a recurrent system to store a bit of information robustly against noise, its dynamics must be contractive around the states that encode the bit — and contractive dynamics are precisely those whose Jacobian products, hence gradients, decay exponentially. Robust storage and trainable long-range credit assignment are in direct tension. The LSTM does not refute the theorem; it routes around it by making the contraction rate a learned, per-dimension quantity (the forget gate), so the model can hold selected dimensions at the knife edge while letting others decay.

Other attacks on vanishing gradients. Gating was not the only idea. One line constrained the recurrent matrix to be orthogonal or unitary (all eigenvalues on the unit circle), making the linear part of the recurrence norm-preserving by construction; these models proved long dependencies were learnable in principle but were fiddly and never displaced LSTMs in practice. Another line (echo-state and reservoir approaches, which your truncated-BPTT experiment accidentally reproduced) skips training the recurrence entirely and only trains the readout, exploiting the short-term memory random dynamics provide for free. Both are worth knowing as conceptual poles: fully constrained recurrence, and untrained recurrence.

Recurrence is being rearchitected, not just retired. Starting around 2023, a family of models revived the recurrent form with the training problem redesigned away. RWKV (Peng et al., 2023) reformulates attention so it can be computed either in parallel across the sequence at training time or as a recurrence at inference time. Mamba (Gu & Dao, 2023) builds on selective state-space models: a state that evolves under input-dependent dynamics (a gate, in LSTM vocabulary) but with a mathematical structure (associative scans) that lets training parallelize across time the way Transformers do, while inference keeps the RNN virtues of constant memory and constant per-token cost. Whether such architectures displace attention at scale is an open empirical question; hybrid stacks interleaving both kinds of layers appear in several recent production models. The scorecard to keep in mind: parallel training was the Transformer’s decisive win, and the new recurrent models match it; constant-memory inference is recurrence’s native win, and Transformers still pay for it in KV-cache (a cost quantified in the next section).

Common pitfalls

  • Loss goes to NaN a few hundred steps into training an RNN. Symptom of exploding gradients: some subsequence pushed the recurrent Jacobian product above 1. Fix: clip gradient norm from the start (e.g. torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)), since clipping cannot repair weights that are already NaN, and consider a smaller learning rate.
  • Model trains but only uses the last few tokens of context. Vanishing gradients: the learning signal from distant tokens decayed to numerical zero, so only short-range patterns get learned. Fix: use an LSTM/GRU rather than a vanilla RNN, initialize the forget-gate bias to a positive value by hand (PyTorch initializes it near 0; 1, in one of PyTorch’s two bias vectors, is the common choice), and keep truncation windows longer than the dependencies you care about.
  • Hidden state carried across batches causes “backward through the graph a second time” errors. Cause: the carried state still references the previous window’s computation graph. Fix: detach the carried state (h, and c for an LSTM) at every truncation boundary — that is what “carry the state, not the gradient” means in code. Passing retain_graph=True instead only trades this error for one about a variable “modified by an inplace operation”, once the optimizer has updated the weights.
  • LSTM underperforms because sequences were padded and the padding was processed. Cause: the cell happily updates its state on pad tokens, contaminating the final hidden state. Fix: use lengths-aware utilities (pack_padded_sequence) or mask the loss and read the state at each sequence’s true last step.

Try it yourself

  1. In the scalar-recurrence script, set w = 1.0 and rerun the tanh part. The linear analysis says gradients should be preserved — what actually happens, and why?
  2. Extend the RNN-vs-LSTM script with a GRU column (nn.GRUCell). Where does it fall between the vanilla RNN and the LSTM at T = 40 and T = 80?
  3. In the LSTM columns, try forget-gate biases of −1 (gates mostly closed) and +3 (almost fully open); the bias the gate sees is the sum of the forget slices of bias_ih and bias_hh. Predict the ordering of gradient norms at T = 80 before you run.
  4. Change the loss from h.sum() on the last state to a sum over all hidden states. Explain why the measured gradient at the first input changes the way it does.
  5. In the truncated-BPTT experiment, set LAG = 8 and rerun the same windows. Predict which windows can now learn the task before running, then check whether the window-8 run behaves like the old window-4 run and explain why it must.
  6. In the clipping experiment, replace the vanilla nn.RNN with nn.LSTM (keep clipping on) and compare the loss trajectory. How much of the remaining difficulty was explosion, and how much was vanishing?

Further reading

  • Hochreiter & Schmidhuber, “Long Short-Term Memory,” Neural Computation, 1997. doi:10.1162/neco.1997.9.8.1735 — the original LSTM paper; the constant-error-carousel analysis in its early sections is still the clearest account of why additive cell updates fix vanishing gradients.
  • Bengio, Simard & Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, 1994. doi:10.1109/72.279181 — the theoretical statement of the storage-versus-trainability dilemma discussed in Going deeper.
  • Pascanu, Mikolov & Bengio, “On the difficulty of training recurrent neural networks,” ICML 2013. arXiv:1211.5063 — the standard analysis of exploding/vanishing gradients, and the paper behind the gradient-clipping recipe you ran above.
  • Sutskever, Vinyals & Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS 2014. arXiv:1409.3215 — LSTM encoder–decoders at the moment they became the state of the art in translation; sets up the bottleneck the next section removes.
  • Schuster & Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing, 1997. doi:10.1109/78.650093 — the original bidirectional architecture; short and readable.
  • Graves, Fernández, Gomez & Schmidhuber, “Connectionist Temporal Classification,” ICML 2006. PDF — the alignment-free sequence loss sketched above, with the dynamic-programming derivation in full.
  • Gu & Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” 2023. arXiv:2312.00752 — the most prominent of the modern recurrent revivals discussed in Going deeper.
  • d2l.ai, Chapter 9: Recurrent Neural Networks and Chapter 10: Modern RNNs (LSTM) — a fuller code-first treatment of everything here, including GRUs and deep/bidirectional variants.
  • PyTorch docs: nn.LSTM — the reference for gate ordering, shapes, and the bidirectional and num_layers flags used in this section’s exercises.

Before class you should be able to

  • Write the vanilla RNN update and explain what the hidden state and weight sharing buy
  • Explain BPTT as backprop on an unrolled network, and what detaching at a truncation boundary does
  • Predict whether a gradient vanishes or explodes from the recurrent weight, using the \(w^{T-1}\) picture
  • State what gradient clipping fixes and what it cannot fix
  • Walk through the LSTM cell update and explain the forget gate’s role in gradient flow
  • Explain why a window no longer than the dependency length makes that dependency unlearnable
  • Name the two RNN limitations (serial computation, long relay paths) that motivated attention

Connects to

  • The S15 review guide covers S01–S13 for self-study (S15 is not held this term); this session is the last one in its scope, so read attentively.
  • Quiz 2 takes the final 30 minutes of this class (6:45–7:15 PM, in Canvas), covering Module 2 (S09–S12). This session is itself covered by Quiz 3, given at S23.
  • HW3 (Transformer from scratch, out at S18) contains no RNN code, but the failure modes measured here are why the architecture you will build there exists.

← S12 · Training in Practice · All readings · S14 · Attention →