Session S02

From Linear Models to MLPs

Activations · Hidden layers · Expressivity

Deep Learning — CS 4715/5715 · University of Idaho

Missing a quiz: the make-up rule

  • Planned absence (travel, a conference, any foreseeable conflict): request at least 1 week before quiz day → an early sitting or make-up is arranged
  • Foreseeable illness or medical treatment: let me know at least 3 days before quiz day → a make-up is arranged
  • Sudden illness or emergency: contact me within 48 hours afterward, with documentation → a make-up is arranged
  • Otherwise: no make-ups, and all four quizzes count — the Online (Web) Wednesday–Sunday window is the built-in flexibility
  • First quiz: S09, covering all of Module 1. Full text in the syllabus and on Canvas.

The two-step move

  • You know linear models: weighted sums, threshold or sigmoid on top
  • Two steps turn them into neural networks: insert a nonlinearity, then stack
  • Skip the nonlinearity → a ten-layer network collapses into the linear model you started with
  • Skip the stacking → you keep the perceptron's blind spots
  • Result: the multilayer perceptron (MLP) — the architecture under everything in this course
  • CNNs are MLPs with shared weights; in a standard Transformer block with a 4×-expanded feed-forward sublayer, the MLP holds roughly two-thirds of the block's non-embedding parameters.

The perceptron and its learning rule

  • Rosenblatt (1958), labels \(y \in \{-1,+1\}\):   \(\hat{y} = \operatorname{sign}(\mathbf{w}^\top\mathbf{x} + b)\)
  • Rule: visit examples one at a time; on a mistake, add \(y_i\,\mathbf{x}_i\) to \(\mathbf{w}\) and \(y_i\) to \(b\); else do nothing
  • Perceptron convergence theorem (early 1960s): if any separating boundary with margin exists, finitely many mistakes suffice — one of the first learning guarantees ever proved
  • The sharp edge: if no separating line exists, the rule never settles — it cycles forever

The convergence proof, in three inequalities

Fold \(b\) into \(\mathbf{w}\) · \(\lVert\mathbf{x}_i\rVert \le R\) · a unit \(\mathbf{w}^{*}\) with \(y_i\,{\mathbf{w}^{*}}^{\!\top}\mathbf{x}_i \ge \gamma\) for all \(i\) · start \(\mathbf{w}_0 = \mathbf{0}\) · \(\mathbf{w}_k\) = weights after the \(k\)-th mistake

  1. Alignment grows linearly: each mistake adds \(y_i\,\mathbf{x}_i^\top\mathbf{w}^{*} \ge \gamma\), so \(\;\mathbf{w}_k^\top\mathbf{w}^{*} \ge k\gamma\)
  2. Length grows like a square root — the middle term is \(\le 0\) precisely because this step was a mistake: \[\begin{aligned}\lVert\mathbf{w}_k\rVert^2 &= \lVert\mathbf{w}_{k-1}\rVert^2 + \underbrace{2\,y_i\,\mathbf{w}_{k-1}^{\top}\mathbf{x}_i}_{\le\,0} + \lVert\mathbf{x}_i\rVert^2\\ &\le \lVert\mathbf{w}_{k-1}\rVert^2 + R^2 \;\Rightarrow\; \lVert\mathbf{w}_k\rVert^2 \le kR^2\end{aligned}\]
  3. Squeeze (Cauchy–Schwarz): \[k\gamma \le \mathbf{w}_k^\top\mathbf{w}^{*} \le \lVert\mathbf{w}_k\rVert \le \sqrt{k}\,R \;\Rightarrow\; k \le (R/\gamma)^2\]

Alignment rises linearly, length only like \(\sqrt{k}\) — the angle to \(\mathbf{w}^{*}\) must keep closing, and it cannot close forever. This slide is the backup for the live derivation.

Both behaviors in one script (reading 1.2)


def train_perceptron(X, y, epochs=50):
    w = np.zeros(X.shape[1]); b = 0.0; history = []
    for epoch in range(epochs):
        mistakes = 0
        for xi, yi in zip(X, y):
            if yi * (w @ xi + b) <= 0:      # wrong side (or on) the boundary
                w += yi * xi; b += yi; mistakes += 1
        history.append(mistakes)
        if mistakes == 0: break
    return w, b, history
      

separable data: mistakes per epoch = [2, 0]
separable data: converged after 2 epochs, w = [3.6008164  3.06362926], b = 0.0
XOR data: ran 1000 epochs, mistakes in the last 5 epochs = [4, 4, 4, 4, 4]

Separable blobs: two mistakes, done. XOR: 1000 epochs later, still wrong on all four points — what it searches for does not exist.

Why the field stalled for ~17 years

  • Minsky & Papert (1969) targeted single-layer perceptrons — and researchers knew multilayer networks were more expressive
  • The real gap: Rosenblatt's mistake-driven rule cannot assign credit to hidden units — nobody knows their "correct" outputs
  • Missing ingredient = a training algorithm, not an architecture
  • Filled by backpropagation's popularization in 1986 (→ S04–S05)
  • Rosenblatt was a psychologist at Cornell Aeronautical Lab; he died in 1971, before multilayer networks were vindicated.

Stacking linear layers is not enough

  • Compose two linear layers: \(\mathbf{h} = W_1\mathbf{x} + \mathbf{b}_1\),  \(\hat{\mathbf{y}} = W_2\mathbf{h} + \mathbf{b}_2\)
  • Then \(\hat{\mathbf{y}} = (W_2 W_1)\,\mathbf{x} + (W_2\mathbf{b}_1 + \mathbf{b}_2)\) — one linear layer in disguise
  • Stack a hundred: still a line; the XOR failure survives intact
  • Cure — the activation function \(f\), elementwise and nonlinear, between layers:
  • \(\mathbf{h} = f(W_1\mathbf{x} + \mathbf{b}_1)\),  \(\hat{\mathbf{y}} = W_2\mathbf{h} + \mathbf{b}_2\) — no longer collapsible
  • Pitfall preview: a "deep" net that performs exactly like logistic regression usually has a missing activation.

Three activations, most of the history

  • Sigmoid \(\sigma(z) = 1/(1+e^{-z})\), range \((0,1)\): still the standard output unit for binary classification; dominated hidden layers in the 1980s–90s
  • Tanh, range (−1, 1): zero-centered rescaled sigmoid; trains a bit better, displaced sigmoid in hidden layers
  • ReLU \(\operatorname{relu}(z) = \max(0, z)\): not differentiable at \(0\), discards negative magnitude — and became the deep-era default
  • Takeover ~2010–2011 (Nair & Hinton; Glorot, Bordes & Bengio): deep rectifier nets train faster and often better
  • AlexNet (2012) used ReLU and credited it with a large training speedup; the field followed.

Why ReLU wins: look at the derivatives (reading 1.2)


z      d_sigmoid  d_tanh   d_relu
 0.0     0.2500   1.0000     0.0
 2.0     0.1050   0.0707     1.0
 5.0     0.0066   0.0002     1.0
10.0     0.0000   0.0000     1.0

product of 20 sigmoid derivatives: 5.77e-17
product of 20 tanh derivatives:    3.41e-13
product of 20 relu derivatives:    0.00e+00

Backprop multiplies one local derivative per layer. Sigmoid's never exceeds 0.25 and dies away from zero → 20 layers give ~1e-17: the vanishing gradient. ReLU's derivative is exactly 1 for \(z > 0\), so the activation adds no decay at any depth — though the weight matrices still can: reading 1.2 measures ReLU at a 1.9e-4 ratio over 12 layers.

ReLU's price: dead units

  • Third line above: derivative is exactly 0 for negative inputs — that single 20-step chain is blocked completely
  • In a real network each weight sits on many paths; gradient flows while some paths are active
  • But a unit whose pre-activation is negative for every input learns nothing: a dead ReLU
  • Usual causes: too-large learning rate, bad initialization
  • Motivates leaky ReLU / GELU — a small slope on the negative side
  • At \(z = 0\) exactly, implementations pick a convention (ours: 0); it does not matter in practice (→ 1.4).

What a hidden layer computes

  • One-hidden-layer MLP: \(\hat{\mathbf{y}} = W_2\, f(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2\)
  • Each hidden unit is a little linear classifier \(f(\mathbf{w}_j^\top\mathbf{x} + b_j)\) — its own hyperplane through input space
  • The hidden layer re-represents the input: "which side of each of the H learned boundaries, and by how much"
  • In that representation, XOR becomes linearly separable (the 52% → 100% jump from S01)
  • Two ReLU units suffice for XOR: "both high," "both low," output fires when neither is active
  • ReLU geometry: every function a ReLU MLP computes is piecewise linear — one crease direction per hidden unit. More units, more creases, finer approximation.

Universal approximation, watched live (reading 1.2)

Cybenko (1989) and Hornik, Stinchcombe & White (1989): one hidden layer, wide enough, approximates any continuous function on a bounded region. Demo: wiggly 1-D target, evenly spaced ReLU kinks, fit only the output layer, double the width:


width   2: max |error| = 1.1074
width   4: max |error| = 0.6698
width   8: max |error| = 0.2268
width  16: max |error| = 0.0567
width  32: max |error| = 0.0148
width  64: max |error| = 0.0038

Once the net has enough kinks: each doubling cuts worst-case error ~4× (piecewise-linear approximation of a smooth curve). Any accuracy you name, some width delivers — and you pay for every digit with more units.

What the theorem does not say

  • Existence only: good weights exist — nothing about whether gradient descent finds them
  • No generalization: a wide-enough network can also fit noise perfectly
  • Width can be ruinous: some functions need polynomially many units deep but provably exponentially many in one layer — depth buys efficiency of representation
  • The most misquoted result in deep learning: it does not say "a neural net can learn anything"
  • Useful check when you want to argue your net can fit the data: overfit a small subset. Ten minutes of experiment beats a theorem that was never about training.

For next time

  • Read: 1.3 Loss Functions and Gradient Descent
  • S03: loss functions and gradient descent; cross-entropy; numerical stability
  • Try it yourself (reading 1.2): hand-construct a 2-hidden-unit ReLU network that computes XOR, and verify it in NumPy.