Session S01

Course Introduction

Why deep learning works now · How this course runs

Deep Learning — CS 4715/5715 · University of Idaho

What this course is

• Implementation-first: every major idea is something you build and run
• You will write an autograd engine from scratch, train networks in PyTorch, implement a Transformer, fine-tune with LoRA, and defend a semester project in person
• Arc: backpropagation → CNNs → Transformers & LLMs → diffusion → efficiency, evaluation, safety
• Framework: PyTorch · a laptop covers the day-to-day work; a free Google Colab account covers the training-heavy parts
• Three sections, one course: on-campus (Moscow), Virtual Meeting (synchronous, over Zoom), Online (Web) asynchronous
• Prereqs: comfortable Python/NumPy + an intro ML course. We start at neural networks.

1958: a machine that learns

• McCulloch & Pitts (1943): networks of threshold units can implement logical functions
• Rosenblatt's perceptron (1958): \(\hat{y} = \operatorname{sign}(\mathbf{w}^\top\mathbf{x} + b)\) — and it learns its weights from labeled examples
• Not just math: the Mark I Perceptron hardware classified images via photocells
• Press promised machines that would walk, talk, see — the first hype cycle
• 1969, Minsky & Papert, Perceptrons: a single layer cannot represent XOR — no line separates the classes
• Overpromise → underdeliver → "AI winter" (mid-1970s, late 1980s). A recurring pattern worth remembering.

The XOR limit — and its cure (reading 1.1)

# XOR-like data: four Gaussian blobs, opposite corners share a label
# --- Model 1: logistic regression (a linear model) ---
p = sigmoid(X @ w + b)
# --- Model 2: same optimizer, same loss, one hidden layer of 8 tanh units ---
H = np.tanh(X @ W1 + b1)
p = sigmoid(H @ w2 + b2)
# (full ~40-line NumPy script in reading section 1.1)
      

linear model  train accuracy: 0.5200
1-hidden-layer train accuracy: 1.0000
      

This run's line lands at 52% — chance. No single line separates XOR's opposite corners — the best any line can do is 3 blobs of 4 — and on this symmetric data gradient descent settles on the nearly flat line instead. One hidden layer → 100%: it learns a representation in which the classes are linearly separable. Learning representations is the course's core idea.

1986–2012: working, but out of fashion

• Backprop popularized by Rumelhart, Hinton & Williams (Nature, 1986): hidden layers become trainable
• 1990s: LeCun's convolutional nets read a significant fraction of U.S. bank checks
• Yet SVMs beat neural nets on most benchmarks of the day; "neural nets" became an unfashionable phrase
• What was missing (not the core idea):
• Compute — serial CPU training; interesting networks took weeks
• Data — labeled sets of thousands; big nets overfit them
• Algorithms — saturating activations → vanishing gradients; initialization was folklore

2012: the ImageNet moment

• GPUs (built for games) turn out nearly ideal for dense matrix multiplies
• ImageNet (Fei-Fei Li's group, 2009): millions of labeled photos + an annual challenge
• Algorithmic fixes accumulate: ReLU, dropout, better initialization, data augmentation
• AlexNet (Krizhevsky, Sutskever, Hinton): ~a week of training on two consumer GPUs
• Top-5 error 15.3% vs 26.2% for second place — a rout in a fractions-of-a-percent sport
• Within ~2 years every serious entry was a deep net; hand-designed features (SIFT, HOG) displaced
• Hinton, Bengio & LeCun shared the 2018 Turing Award — for decades of unfashionable persistence.

The triad, and napkin math (reading 1.1)

Compute × data × algorithms — none sufficient alone, all still compounding. Habit to build: count params and FLOPs before training. Linear layer: \(n_{\text{in}}\,n_{\text{out}}\) weights, ~\(2\,n_{\text{in}}\,n_{\text{out}}\) FLOPs; a training step ≈ 3× the forward pass.

logistic regression  [784, 10]                  7,850 params      15,690 FLOPs/example
small MLP            [784, 256, 10]           203,530 params     406,794 FLOPs/example
wider + deeper MLP   [784, 1024, 1024, 10]  1,863,690 params   3,725,322 FLOPs/example

training the deeper net, 10 epochs on 60k examples: ~6.71e+12 FLOPs
at an assumed 10^12 FLOP/s: ~6.7 s of pure arithmetic
      

Order of magnitude, not a stopwatch: 10^12 FLOP/s is an assumed rate (above one CPU core) and counts arithmetic only. The point — at this size, compute is not the obstacle. Same arithmetic, bigger exponents: AlexNet ~60M params; frontier LLMs → months on thousands of GPUs.

What "deep" buys — and what it costs

• Buys: composition (edges → textures → parts → objects); some functions need exponentially many units in one layer; learned features transfer (basis of fine-tuning, HW2+)
• Costs — the honest ledger:
• Data hunger: with a few hundred examples and no pretraining, linear/tree baselines often win
• Brittleness: spurious correlations; adversarial perturbations (Szegedy et al., ICLR 2014)
• Opacity, compute cost, and no guarantees — non-convex optimization, disciplined empiricism
• Project policy follows: a careful experiment where the simple baseline wins is a successful project.

Grading

A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60 (cutoffs may be lowered, never raised). No midterm exam, no final exam — the four module quizzes replace them. CS 4715 sums to 100%; CS 5715 sums to 110% (papers raise the ceiling) — bonus can add up to 10 more to either.

Quizzes and the late policy

• Module quizzes: four, each the final 30 min of a Wednesday class (6:45–7:15 PM), ~10 questions, same bank for everyone
• Every section in Canvas · Online (Web) and approved flexible-time arrangements: your assigned window
• No late quizzes; make-ups in three cases only — planned absence ≥ 1 week ahead · foreseeable illness ≥ 3 days · sudden illness/emergency: within 48 h after, documented
• Late policy for dated submissions (HW, project docs):
• 12-hour grace period, no penalty → then −10% of max score per 24 h (or fraction)
• Penalties floor at 60% credit for anything submitted by the end of the S29 week
• Regrade requests: in writing, within 7 days of the grade posting.

AI use: three tiers

Why the difference? Homework builds skills — outsourcing it shows up on quizzes and the code defense. The project is research — there, modern tools are part of the craft, used transparently. You must be able to explain any line you turn in.

Homework and the compute promise

• Four tested Python projects: starter repo, documented stubs, pytest suite; autograding on every push
• HW1 micro-autograd → HW2 CNNs/transfer → HW3 Transformer → HW4 evaluation/efficiency
• Required: a laptop with Python + a free Google Colab account. Not required: paid compute, a personal GPU.
• Homework test suites run on CPU in seconds — daily feedback needs only your laptop
• Training parts of some HWs and most projects need a GPU — the free Colab tier is sufficient and expected
• Course theme: get signal from small, well-designed experiments. Semester project: teams of 1–3 (5715: max 2), any DL problem doable on free compute.

For next time

• Read: 1.1 Why Deep Learning Works Now (today's session) and 1.2 From Linear Models to MLPs (next session)
• S02: from linear models to multilayer perceptrons — activations, expressivity
• Mark your calendar: HW1 (micro-autograd) goes out at S04. The first quiz is at S09 and covers all of Module 1.
• Set up now: Python + NumPy environment, GitHub account, Canvas access. Office hours posted on Canvas.