Module 1 · Reading 1.1 · read before S01: Course Introduction

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

Every tool you will use in this course (automatic differentiation, GPUs, pretrained models) exists because a family of methods that barely worked for fifty years suddenly started working extremely well. Understanding why that happened is not just history for its own sake. The same three ingredients that flipped the switch around 2012 are still the levers that decide whether your own projects succeed: compute, data, and a handful of algorithmic fixes. When a model underperforms in this course, your diagnosis will almost always come down to one of them.

This section tells the story of neural networks from the 1940s to the present, and uses four small experiments to make the story concrete: one showing exactly what a hidden layer buys you, one showing what the field did instead of hidden layers for decades, one showing how to reason about the cost of a model before you train it, and one measuring what more data is actually worth.

How to read this

The perceptron era: a machine that learns

The idea that computation could be built from neuron-like units goes back to Warren McCulloch and Walter Pitts, who showed in 1943 that networks of simple threshold units can implement logical functions. Donald Hebb’s 1949 book The Organization of Behavior added the complementary idea that learning might consist of strengthening connections between units that fire together. But the field’s real starting gun was Frank Rosenblatt’s perceptron (1958): a model that computes a weighted sum of its inputs, outputs a class based on the sign, and, crucially, learns its weights from labeled examples.

Rosenblatt did not just publish the mathematics. His group at the Cornell Aeronautical Laboratory, funded by the U.S. Office of Naval Research, built the idea into hardware: the Mark I Perceptron, a room-filling analog machine whose “retina” was a 20-by-20 grid of 400 photocells, and whose learnable weights were physical potentiometers adjusted by electric motors during training. Show it labeled examples of simple shapes, and the motors would grind the weights toward a configuration that classified them. Every w -= lr * grad you write in this course is the software descendant of those motors.

A perceptron is what we would today call a linear classifier. It predicts

\[\hat{y} = \operatorname{sign}(\mathbf{w}^\top \mathbf{x} + b)\]

and nudges \(\mathbf{w}\) whenever it makes a mistake. That is the entire model. Its significance was less the architecture than the demonstration that a machine could improve from experience — a genuinely new capability in 1958, and one the press covered with spectacular enthusiasm. Contemporary newspaper accounts promised machines that would soon see, talk, and be conscious of their own existence. The gap between that promise and what a linear classifier can actually do set the field up for its first crash.

Historical note. After a 1958 press conference, The New York Times reported that the perceptron was expected to eventually walk, talk, see, and reproduce itself. Overpromising followed by underdelivering became a recurring pattern: the field has been through at least two “AI winters” (periods of collapsed funding and credibility, roughly the mid-1970s and late 1980s), each preceded by exactly this kind of hype cycle. It is worth remembering that today’s claims about deep learning will be graded by the same historians.

The crash had a technical trigger. In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a careful mathematical analysis showing that single-layer perceptrons cannot represent certain simple functions — most famously XOR, the function that is true when exactly one of two inputs is true. No line can separate the XOR-positive points from the XOR-negative points, so no linear model can learn it, no matter how long you train. The book is often credited (by most accounts, somewhat unfairly to its authors’ intent) with redirecting funding away from neural networks for a decade.

Here is the limitation, and its cure, in about forty lines of NumPy. We generate an XOR-like dataset (four clusters where opposite corners share a label) and train two models with plain gradient descent: a linear classifier (logistic regression) and the same classifier with one hidden layer of 8 tanh units inserted before it.

import numpy as np

np.random.seed(0)

# XOR-like data: four Gaussian blobs, opposite corners share a label
centers = np.array([[1, 1], [-1, -1], [1, -1], [-1, 1]], dtype=float)
labels = np.array([0, 0, 1, 1])  # (1,1) and (-1,-1) -> class 0; the others -> class 1
X = np.vstack([c + 0.3 * np.random.randn(50, 2) for c in centers])   # (200, 2)
y = np.repeat(labels, 50).astype(float)                              # (200,)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def accuracy(p, y):
    return np.mean((p > 0.5) == y)

# --- Model 1: logistic regression (a linear model) ---
w, b = np.zeros(2), 0.0
lr = 0.5
for step in range(2000):
    p = sigmoid(X @ w + b)                 # predictions, (200,)
    grad_z = (p - y) / len(y)              # dL/dz for cross-entropy loss
    w -= lr * (X.T @ grad_z)
    b -= lr * grad_z.sum()
print(f"linear model  train accuracy: {accuracy(sigmoid(X @ w + b), y):.4f}")

# --- Model 2: one hidden layer of 8 tanh units ---
rng = np.random.default_rng(0)
W1 = 0.5 * rng.standard_normal((2, 8)); b1 = np.zeros(8)
w2 = 0.5 * rng.standard_normal(8);      b2 = 0.0
for step in range(2000):
    H = np.tanh(X @ W1 + b1)               # hidden activations, (200, 8)
    p = sigmoid(H @ w2 + b2)
    grad_z = (p - y) / len(y)              # (200,)
    grad_w2 = H.T @ grad_z
    grad_H = np.outer(grad_z, w2) * (1 - H**2)   # backprop through tanh
    W1 -= lr * (X.T @ grad_H); b1 -= lr * grad_H.sum(axis=0)
    w2 -= lr * grad_w2;        b2 -= lr * grad_z.sum()
H = np.tanh(X @ W1 + b1)
print(f"1-hidden-layer train accuracy: {accuracy(sigmoid(H @ w2 + b2), y):.4f}")

Running this prints:

linear model  train accuracy: 0.5200
1-hidden-layer train accuracy: 1.0000

The linear model lands at 52% on this run — essentially a coin flip. That number is what this particular fit reached, not a proven ceiling; what is guaranteed is the shape of the failure. XOR puts the two classes on opposite corners, so no single straight line has them on opposite sides of it: whichever line you draw, at least one of the four blobs lands on the wrong side, so the best any line can do here is three blobs out of four, about 75%. Gradient descent does not even find that line. The cross-entropy loss punishes the one confidently wrong blob more than it rewards the three right ones, so on this symmetric data the loss-minimizing line is the nearly flat one (\(\mathbf{w} \approx 0\), predicting about 50/50 everywhere), and that is where training settles however long it runs. The 52% is chance plus tie-breaking. Adding one hidden layer takes the same optimizer, the same data, and the same loss to 100%. The hidden layer learns a new representation of the input in which the classes are linearly separable; the output layer then only has to draw a line. This is the core idea the rest of the course elaborates: learning representations instead of hand-designing them. Don’t worry about the two grad_ lines yet; deriving them is the business of section 1.4.

The feature-engineering detour

There is a second cure for XOR, and the field spent roughly forty years on it: instead of letting the model learn a representation, hand it a better one. If someone tells you the classes are defined by “opposite corners agree,” you can compute the product x1 * x2 as a third input feature (positive for the corners (1,1) and (-1,-1), negative for the other two), and a linear model over the augmented features solves the problem instantly:

import numpy as np

np.random.seed(0)

# Same XOR-like data as the first experiment
centers = np.array([[1, 1], [-1, -1], [1, -1], [-1, 1]], dtype=float)
labels = np.array([0, 0, 1, 1])
X = np.vstack([c + 0.3 * np.random.randn(50, 2) for c in centers])
y = np.repeat(labels, 50).astype(float)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def train_logreg(F, y, lr=0.5, steps=2000):
    """Logistic regression by gradient descent on feature matrix F."""
    w, b = np.zeros(F.shape[1]), 0.0
    for _ in range(steps):
        p = sigmoid(F @ w + b)
        g = (p - y) / len(y)
        w -= lr * (F.T @ g)
        b -= lr * g.sum()
    acc = np.mean((sigmoid(F @ w + b) > 0.5) == y)
    return acc, w

feature_sets = {
    "raw features [x1, x2]":        X,
    "engineered [x1, x2, x1*x2]":   np.column_stack([X, X[:, 0] * X[:, 1]]),
}
for name, F in feature_sets.items():
    acc, w = train_logreg(F, y)
    print(f"{name:<28} accuracy = {acc:.4f}   weights = {np.round(w, 2)}")

Running this prints:

raw features [x1, x2]        accuracy = 0.5200   weights = [-0.04 -0.07]
engineered [x1, x2, x1*x2]   accuracy = 1.0000   weights = [ 0.31 -0.01 -8.91]

Same linear model, same optimizer: 52% on raw coordinates, 100% with one extra feature. Look at the learned weights — nearly all the work is done by the x1*x2 feature (weight −8.91), and the raw coordinates are nearly ignored. The model did not get smarter; the input representation got smarter, because a human who already understood the problem baked the answer into a feature.

This is not a toy observation; it describes mainstream machine learning from the 1970s into the 2000s. Practitioners engineered features by hand and fed them to well-understood shallow learners: edge detectors and gradient histograms for images (SIFT, HOG), cepstral coefficients for speech, n-grams for text. Kernel methods, which took off after Cortes and Vapnik’s support-vector network paper (Machine Learning, 1995), gave the strategy its most elegant form: a kernel implicitly maps inputs into a high-dimensional feature space chosen by the algorithm designer. The approach won because it worked, and its limits took a long time to bite: every new domain needed its own years of feature craftsmanship, and the representation stayed fixed as more data arrived even though the fitted predictor could still improve. Performance therefore plateaued when the hand-designed representation stopped exposing useful structure. The deep learning bet, unfashionable for decades, was that features should be learned from data like everything else. Keep both cures for XOR in mind as Module 1 proceeds: the hidden layer discovered a product-like feature on its own; the engineer had to know it in advance.

The backprop era: training deep networks, slowly

Minsky and Papert’s critique applied to single-layer networks. Networks with hidden layers escape it — but in 1969 nobody had a practical way to train the hidden weights. The missing piece was backpropagation: apply the chain rule through the network to compute how the loss changes with respect to every weight, then adjust all weights by gradient descent. The algorithm has several independent discoverers (we take up that history in section 1.4), but the paper that made the field pay attention was Rumelhart, Hinton, and Williams’ 1986 Nature paper, which showed that networks trained this way learn useful internal representations.

The late 1980s and 1990s produced real successes. Yann LeCun and colleagues trained convolutional networks to read handwritten digits, and by the late 1990s their system was reading a significant fraction of bank checks deposited in the United States. Yet neural networks remained a minority taste. They were finicky to train, theory lagged behind practice, and by the mid-1990s methods with stronger guarantees and less tuning, support vector machines in particular, outperformed them on most benchmark tasks of the day. A graduate student around 2000 faced a clear choice: kernel machines came with convex optimization, a convergence theory, and reproducible results; neural networks came with folklore about learning rates and initializations, and results that varied run to run. Most chose accordingly.

Through the 2000s, “neural nets” was, by most accounts, an unfashionable phrase on grant applications; two 2006 papers from Hinton’s group (the deep-belief-network paper with Osindero and Teh, and a Science paper with Salakhutdinov on deep autoencoders) helped rebrand the enterprise as “deep learning,” but the results that would convince skeptics had not arrived yet.

It is worth being precise about what was missing, because none of it was the core idea:

  • Compute. Training was serial, on CPUs. Networks big enough to be interesting took weeks, so the iteration loop of research was glacial.
  • Data. Labeled datasets were tiny — thousands of examples. Big networks overfit them badly, so small, heavily engineered models won.
  • Algorithms. Sigmoid-style activations saturate, making gradients vanish in deep networks (section 1.2); initialization and regularization were folklore rather than method.

2012: the ImageNet moment

All three gaps closed within a few years of each other, and it is worth watching each one close.

The data gap closed first, and deliberately. Fei-Fei Li’s group set out around 2007 to build an image dataset at a scale the field considered pointless, organizing it around the WordNet hierarchy of noun categories and using crowdsourced workers on Amazon Mechanical Turk to verify labels. The resulting ImageNet paper (Deng et al., CVPR 2009) described a collection already in the millions of images. Attached to it from 2010 was an annual competition, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), whose classification task gave models roughly 1.2 million training images across 1,000 categories. For the first time, a model with tens of millions of parameters had somewhere to earn its keep.

People. Fei-Fei Li, then a new professor at Princeton and later at Stanford, has recounted that ImageNet was widely dismissed while it was being built — the mainstream view held that better algorithms, not more data, were the path forward, and dataset-building was low-prestige work. The project labeled millions of images anyway, at a scale only made feasible by crowdsourcing. The 2012 result vindicated the bet publicly, and the lesson (benchmark datasets can redirect an entire field) has shaped how research communities operate since.

The compute gap closed by accident. Graphics processors, built to apply the same arithmetic to millions of pixels for video games, turned out to be nearly ideal for the dense matrix multiplications that dominate neural network training. And on the algorithms side, small fixes with outsized effects accumulated: the ReLU activation, which does not saturate for positive inputs; dropout; better initialization; data augmentation.

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered the challenge with a deep convolutional network, now universally called AlexNet, trained for about a week on two consumer GPUs. The paper reports a top-5 error rate of 15.3%, against 26.2% for the second-place entry built on hand-engineered features. Competition margins are normally measured in fractions of a percentage point; this was a rout, and the field reacted accordingly. Within roughly two years, essentially every serious entry in the competition was a deep network, and computer vision pipelines built on hand-designed features (SIFT, HOG), the accumulated craft of a generation of researchers, were displaced by learned features.

The aftermath moved fast. The ILSVRC organizers’ own retrospective (Russakovsky et al., IJCV 2015) reports that a trained human annotator achieved about 5.1% top-5 error on the same task; the 2015 winning entry, Microsoft Research’s 152-layer residual network, reported 3.57%. In three years the benchmark went from “machines far behind humans” to “machines past a trained human on this specific task” — with the usual caveat that the task (choosing among 1,000 fixed categories, many of them fine-grained dog breeds) measures something narrower than “seeing.” The same pattern then repeated in speech recognition, machine translation, and eventually, via Transformers, in language modeling at large.

People. Geoffrey Hinton, Yoshua Bengio, and Yann LeCun shared the 2018 ACM Turing Award, computing’s highest honor, for their work on deep learning, much of it done during the decades when the approach was out of fashion. All three kept working on neural networks through the 1990s and 2000s with limited mainstream support, a reminder that the “sudden” 2012 breakthrough rested on thirty years of unfashionable persistence. Krizhevsky and Sutskever, the other two authors of AlexNet, were Hinton’s students at Toronto.

The triad: compute, data, algorithms

The honest summary of “why now” is that none of the three ingredients was sufficient alone, and all three kept compounding after 2012. Training compute for landmark models grew far faster than Moore’s law through the 2010s — one widely cited OpenAI analysis estimated a doubling time of a few months between 2012 and 2018. Datasets grew from a million labeled images to trillions of tokens of web text. And algorithmic improvements (normalization, residual connections, attention, better optimizers) repeatedly made bigger models trainable at all.

Historical note. The GPU’s role in deep learning is a story of borrowed hardware. Graphics chips were commodity products aimed at gamers; what made them usable for science was NVIDIA’s release of CUDA in 2007, a programming model that let researchers write non-graphics code for them. By 2009, Raina, Madhavan, and Ng showed at ICML that GPUs could train unsupervised models with on the order of 100 million parameters in a day (work that had been taking weeks on CPUs), and AlexNet’s two consumer gaming cards were the same idea carried to its conclusion. Deep learning did not get its own hardware until it had already succeeded on someone else’s; purpose-built accelerators (TPUs and their kin) came after the breakthrough, not before.

You should develop the habit of thinking about models the way the field does: in parameter counts and FLOPs (floating-point operations). The arithmetic is straightforward enough to do on a napkin, and doing it before training is how you avoid designing a model you cannot afford to run. For a fully connected layer with n_in inputs and n_out outputs, the weight matrix has n_in * n_out entries, and computing the output takes about 2 * n_in * n_out FLOPs (one multiply and one add per weight). A useful rule of thumb: a training step costs roughly three times the forward pass, because the backward pass does about twice the work of the forward pass.

import numpy as np

def mlp_stats(sizes):
    """Parameter count and forward-pass FLOPs for a fully connected net.

    Each layer computes W @ x + b: (in*out) multiplies + (in*out) adds,
    so ~2*in*out FLOPs per layer per example (bias and activation are
    lower-order terms we fold in as +out each).
    """
    params, flops = 0, 0
    for n_in, n_out in zip(sizes[:-1], sizes[1:]):
        params += n_in * n_out + n_out          # weights + biases
        flops += 2 * n_in * n_out + n_out       # multiply-adds + bias adds
    return params, flops

nets = [
    ("logistic regression", [784, 10]),
    ("small MLP",           [784, 256, 10]),
    ("wider + deeper MLP",  [784, 1024, 1024, 10]),
]
for name, sizes in nets:
    p, f = mlp_stats(sizes)
    print(f"{name:<20} {str(sizes):<22} {p:>9,d} params  {f:>10,d} FLOPs/example")

# Training cost = forward + backward (~2x forward) per example, per epoch.
p, f = mlp_stats([784, 1024, 1024, 10])
train_flops = 3 * f * 60_000 * 10               # 3x forward cost, 60k examples, 10 epochs
print(f"\ntraining the deeper net, 10 epochs on 60k examples: ~{train_flops:.2e} FLOPs")
print(f"at an assumed 10^12 FLOP/s: ~{train_flops / 1e12:.1f} s of pure arithmetic")

Running this prints:

logistic regression  [784, 10]                  7,850 params      15,690 FLOPs/example
small MLP            [784, 256, 10]           203,530 params     406,794 FLOPs/example
wider + deeper MLP   [784, 1024, 1024, 10]  1,863,690 params   3,725,322 FLOPs/example

training the deeper net, 10 epochs on 60k examples: ~6.71e+12 FLOPs
at an assumed 10^12 FLOP/s: ~6.7 s of pure arithmetic

Two lessons from those numbers. First, scale is cheap at this end: the entire training run is about 7e12 floating-point operations, which is a small number by modern standards. Read the seconds figure as an order of magnitude, not a stopwatch. The \(10^{12}\) FLOP/s in the last line is a round number we assumed to make the division easy (it is small-accelerator territory, well above what a single CPU core sustains), and the estimate counts only arithmetic. A real run also pays for memory traffic, framework overhead, and data loading, which at this size often cost more than the multiply-adds do. So the honest conclusion is not “this takes 6.7 seconds”; it is “the arithmetic here is nowhere near the obstacle.” Every homework’s automated tests run on CPU; assignments with substantive training, especially transfer learning and LoRA, also provide a free Google Colab GPU path as stated in the syllabus. Second, the numbers grow multiplicatively — AlexNet had about 60 million parameters (the paper’s own count), and modern language models have hundreds of billions, trained on trillions of tokens. The same napkin math that puts our MLP at seconds of arithmetic puts a frontier model at months on thousands of GPUs. The reasoning is identical; only the exponents change.

Data: measuring what more examples buy

The data leg of the triad deserves its own experiment, because “more data helps” is a claim you can quantify on a laptop. We generate a two-class problem with genuine structure (two interleaved spirals with label noise), hold out one large fixed test set, and train the same small network on training sets of growing size:

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
np.random.seed(0)

def make_spirals(n_per_class, rng, noise=0.25):
    """Two interleaved spirals -- a nonlinear problem with a fixed Bayes rate."""
    X, y = [], []
    for c in range(2):
        t = rng.uniform(0.25, 3.0, size=n_per_class)
        ang = 2.5 * t + np.pi * c
        pts = np.stack([t * np.cos(ang), t * np.sin(ang)], axis=1)
        pts += noise * rng.standard_normal(pts.shape)
        X.append(pts)
        y.append(np.full(n_per_class, c))
    return np.vstack(X).astype(np.float32), np.concatenate(y)

def train_mlp(X, y, steps=2000, lr=0.01):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 64), nn.ReLU(),
                          nn.Linear(64, 64), nn.ReLU(),
                          nn.Linear(64, 2))
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    Xt, yt = torch.from_numpy(X), torch.from_numpy(y)
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.functional.cross_entropy(model(Xt), yt)
        loss.backward()
        opt.step()
    return model

# One fixed, large test set; training sets of growing size, same distribution
X_test, y_test = make_spirals(2000, np.random.default_rng(12345))

for n in [25, 100, 400, 1600]:
    X_tr, y_tr = make_spirals(n, np.random.default_rng(0))
    model = train_mlp(X_tr, y_tr)
    with torch.no_grad():
        acc_tr = (model(torch.from_numpy(X_tr)).argmax(1).numpy() == y_tr).mean()
        acc_te = (model(torch.from_numpy(X_test)).argmax(1).numpy() == y_test).mean()
    print(f"n_train = {2 * n:>5}: train acc {acc_tr:.4f}   test acc {acc_te:.4f}")

Running this prints:

n_train =    50: train acc 1.0000   test acc 0.9200
n_train =   200: train acc 1.0000   test acc 0.9585
n_train =   800: train acc 0.9975   test acc 0.9685
n_train =  3200: train acc 0.9831   test acc 0.9740

Read the two columns against each other. With 50 examples the network memorizes its training set perfectly (100% train accuracy) but transfers imperfectly (92% test) — the gap between the columns is overfitting, and it is the small-data disease. Each quadrupling of the data shrinks the gap and lifts test accuracy: 92.0%, 95.9%, 96.9%, 97.4%. Two further things are worth noticing. The returns diminish (the first quadrupling buys 3.9 points, the last buys half a point) because test accuracy is approaching the noise ceiling built into the labels; no amount of data pushes past irreducible label noise. And the gains come without touching the model: same architecture, same optimizer, same step count. When someone says data was a co-equal ingredient in the deep learning breakthrough, this table, scaled up nine orders of magnitude, is the mechanism they mean. The smooth, roughly power-law improvement of loss with dataset size turns out to hold across many domains and many orders of magnitude, and making that observation precise is where the scaling-laws story of section 3.5 begins.

What “deep” buys — and what it costs

Why stack layers at all, rather than making one hidden layer wider? The pragmatic answer: composition. Each layer transforms the representation produced by the previous one, so a deep network can build complicated functions as pipelines of simple stages — edges, then textures, then parts, then objects, in a vision network. Some functions that a deep network represents compactly require exponentially many units to represent with a single hidden layer (section 1.2 demonstrates a version of this). And learned features transfer: the early layers of a network trained on one task are routinely reused for another, which is the basis of the fine-tuning workflow you will use from HW2 onward.

Honesty requires the other column of the ledger:

  • Data hunger. Deep networks are at their best with abundant data. With a few hundred examples and no pretrained starting point, a linear model or gradient-boosted trees will often beat a network you train from scratch — our spiral experiment’s 50-example row hints at why.
  • Brittleness. Networks latch onto whatever correlations predict the training labels, including spurious ones (backgrounds, watermarks, artifacts). Adversarially chosen, imperceptible input perturbations can flip predictions — a phenomenon reported by Szegedy and colleagues in a paper published at ICLR 2014 and still not fully solved.
  • Opacity. A trained network is millions of coupled coefficients. Interpretability research is active, but you should not expect a crisp answer to “why did it predict that?”
  • Cost. Training frontier models consumes enormous compute, energy, and capital, concentrating that capability in a handful of organizations.
  • Limited global guarantees. Optimization is non-convex, so general guarantees that a practical training run reaches a global optimum are scarce. Important results do exist for restricted models, regimes, and algorithms; for full modern systems, disciplined empiricism with controls and error bars remains essential.

Pitfall. “Deep learning won, so use a deep network” is a bad syllogism, and acting on it is the most common mistake in project work. On small tabular datasets, against a tuned linear or tree baseline, a from-scratch neural network frequently loses — and if you never ran the baseline, you cannot know. In this course’s projects, a careful experiment where the simple baseline wins is a successful project; an uncontrolled experiment where the network “wins” is not.

Going deeper

The compute-data-algorithms triad is the standard story, and it is true as far as it goes. Two influential essays sharpen it into something less comfortable.

The bitter lesson. Richard Sutton’s 2019 essay of that name surveys seventy years of AI and argues that the pattern of 2012 is the only pattern: general-purpose methods that improve automatically with more computation (search and learning) have repeatedly and eventually beaten approaches built on human domain knowledge, in chess, Go, speech, and vision alike. The “bitter” part is that building knowledge in feels productive and scientifically respectable, works in the short term, and loses in the long term. Our feature-engineering experiment is the miniature: the x1*x2 feature is clever, correct, and a dead end — it encodes the answer to exactly one problem, while the hidden layer’s discovering of it scales to problems nobody has analyzed yet. The essay is four paragraphs long and worth rereading once a year; it is also worth arguing with, since “add compute” is not actionable advice for a student with one GPU, and the essay is silent on the algorithmic fixes (ReLU, initialization, normalization) that were needed before compute could be absorbed at all.

The hardware lottery. Sara Hooker’s essay (published in Communications of the ACM, 2021) supplies the structural counterpoint: research ideas succeed or fail partly on whether they happen to fit the hardware and software of their era — they win or lose a lottery they did not enter. Neural networks themselves are her leading example: the ideas of the 1980s were not wrong, they were expensive in exactly the operations that CPUs of the time did slowly, and they became “right” when a gaming-driven hardware trajectory delivered cheap parallel matrix arithmetic. The unsettling implication runs forward, too: today’s accelerators are optimized for dense matrix multiplication, so ideas that need something else (sparsity, irregular memory access, very different arithmetic) now face the same headwind neural networks once did. The triad’s “compute” leg, in other words, is not neutral infrastructure; it quietly votes on which algorithms get to look good.

Both essays are readable in one sitting and both appear in Further reading. Between them they turn this section’s history into two questions worth carrying through the semester: what in today’s method stack is a genuine idea, and what is an artifact of what current hardware happens to make cheap? — and, for any technique you learn, does it get better on its own as compute and data grow, or does it cap out at what its designer knew?

A final calibration on the word “suddenly.” The switch did not flip in 2012 so much as become visible then: GPU-trained networks had been setting records in smaller communities for several years (handwriting recognition and traffic-sign benchmarks among them), the algorithmic pieces accumulated one paper at a time across 2006–2011, and the dataset that made the demonstration possible was a five-year construction project. Watersheds in this field tend to be the moment accumulated change becomes impossible to ignore, not the moment it happens — a distinction worth remembering the next time a capability appears to arrive overnight.

Common pitfalls

  • Symptom: your model performs at chance no matter how long you train. Cause: the classes may not be separable in the representation your model can express — the linear-model-on-XOR situation. Fix: check model capacity first: can it overfit a tiny subset (10–50 examples)? If not, the model, not the training loop, is the bottleneck.
  • Symptom: a training run you expected to take minutes takes days. Cause: nobody did the FLOP arithmetic before pressing enter. Fix: estimate 3 * forward FLOPs * examples * epochs before training; if the number divided by your hardware’s throughput is unacceptable, shrink the model or the data first.
  • Symptom: a deep network loses to logistic regression on your dataset. Cause: not a bug — probably too little data for the capacity, and no pretrained features to start from. Fix: treat the linear model as the baseline to beat, use pretraining/transfer where possible, and report the comparison honestly.
  • Symptom: perfect training accuracy, mediocre test accuracy. Cause: overfitting — the model memorized the training set, as in our 50-example spiral run. Fix: more data if you can get it, regularization (section 1.7) if you cannot; and always judge by held-out data.
  • Symptom: results reported in a 2012-era paper seem weak next to what you can do on a laptop today. Cause: the triad kept compounding — you have orders of magnitude more compute and better algorithmic defaults than they did. Fix: when comparing to older work, compare compute budgets too, not just scores.

Try it yourself

  1. In the XOR experiment, reduce the hidden layer from 8 units to 2, then to 1. At what width does the network stop reaching 100% train accuracy? Two hidden units are enough in principle — do they work with this initialization every time? Try a few seeds.
  2. Move the four cluster centers so that opposite corners no longer share a label (e.g., left pair class 0, right pair class 1). Rerun both models and explain the linear model’s new accuracy.
  3. In the feature-engineering experiment, replace the x1*x2 feature with x1^2 and with |x1 - x2|. Which engineered features solve XOR and which do not? State the general property a single added feature needs.
  4. Extend mlp_stats to report bytes of memory for parameters in float32, and estimate for the [784, 1024, 1024, 10] net how many examples per second one training step could process at the same assumed \(10^{12}\) FLOP/s.
  5. Using the rule of thumb 3 * forward FLOPs per training example, estimate the total FLOPs to train a 60-million-parameter network for one epoch on 1.2 million images, assuming forward cost ~2 * params FLOPs per image (a loose lower bound that ignores weight sharing). Compare with the seconds-scale number from our MLP.
  6. In the spiral experiment, hold the training set fixed at 200 examples and instead scale the model: hidden widths 4, 16, 64, 256. Does more capacity help, hurt, or neither at this data size? Reconcile what you see with the data-scaling table.

Further reading

  • Krizhevsky, Sutskever & Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS 2012 — the AlexNet paper: architecture, the two-GPU training setup, and the 15.3% result that ended the feature-engineering era. papers.nips.cc
  • Rumelhart, Hinton & Williams, “Learning representations by back-propagating errors,” Nature, 1986 — the paper that gave hidden layers a training algorithm and the field its second act. nature.com
  • LeCun, Bengio & Hinton, “Deep learning,” Nature, 2015 — a survey of the field by its Turing-Award trio, written just after the breakthrough years; the best single overview of what changed and why. nature.com
  • Deng, Dong, Socher, Li, Li & Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR 2009 — how the dataset was actually built: WordNet structure, crowdsourced verification, and the case for scale. ieee.org
  • Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” IJCV 2015 — the competition’s own retrospective, including the human-performance study cited in this section. arxiv.org/abs/1409.0575
  • Raina, Madhavan & Ng, “Large-scale Deep Unsupervised Learning using Graphics Processors,” ICML 2009 — the early demonstration that GPUs turn weeks of training into a day. dl.acm.org
  • Sutton, “The Bitter Lesson,” 2019 — a short, influential essay arguing that general methods that leverage computation beat human-knowledge-engineered methods; read it alongside this section’s Going deeper. incompleteideas.net
  • Hooker, “The Hardware Lottery,” Communications of the ACM, 2021 — why research ideas win or lose on hardware compatibility, with neural networks as the central case study. cacm.acm.org
  • Dive into Deep Learning, Chapter 1 (Introduction) — a complementary telling of this history with runnable notebooks. d2l.ai

Before class you should be able to

  • Name the three ingredients that converged around 2012 and give one concrete example of each gap closing.
  • Explain why no single line separates XOR-like data, why gradient descent on a linear model nevertheless ends up near chance there rather than at the best line, and what a hidden layer changes about the representation.
  • Contrast learned features with hand-engineered features and state which one improves as data grows.
  • Estimate the parameter count and forward-pass FLOPs of a small fully connected network, and apply the ~3x-forward rule for training cost.
  • Read a train-vs-test accuracy table and point to the overfitting gap and the diminishing returns from more data.
  • List the costs on the other side of the ledger (data hunger, brittleness, opacity, cost, no guarantees) and say when a simple baseline should be expected to win.

Connects to

  • This session is covered by Quiz 1 (Module 1: S01–S08), given at S09.
  • The napkin FLOP math is a habit you will need for compute budgeting in the semester project (project guide out at S11).
  • The capacity experiment previews section 1.2 (S02), and the triad returns explicitly in the scaling-laws session (S18) and the efficiency session (S26).

All readings · S02 · From Linear Models to MLPs →