Session S08

PyTorch in Practice

Turn tensors, modules, data, and an optimizer into one reliable program.

Deep Learning · CS 4715/5715 · University of Idaho

Admin: HW1 is due today

• Due today at 11:59 PM Pacific; a 12-hour grace period follows.
• After the grace period: −10% of the maximum score per 24 hours, down to a 60% floor.
• Run python -m pytest tests -v before uploading the code file.

No new homework today. HW2 is released in S12.

One training program has four connected responsibilities

Each object owns one responsibility; the loop connects them.

A tensor carries more than numbers

For class labels, use torch.long. For model inputs and weights, float32 is the usual default.

Matrix notation predicts the tensor shapes

Let \(X\in\mathbb R^{N\times D}\), \(W\in\mathbb R^{D\times C}\), and \(b\in\mathbb R^C\).

\[Z=XW+b\in\mathbb R^{N\times C}\]

torch.manual_seed(0)
x = torch.randn(4, 3)       # N=4, D=3
w = torch.randn(3, 2)       # D=3, C=2
b = torch.randn(2)          # broadcast over N
z = x @ w + b               # (4,3) @ (3,2) + (2,)
print(x.shape, w.shape, b.shape, z.shape)
  

torch.Size([4, 3]) torch.Size([3, 2]) torch.Size([2]) torch.Size([4, 2])
  

@ contracts the shared \(D\) axis; adding \(b\) broadcasts it across the \(N\) rows.

A scalar loss creates a backward route to every parameter

Given: \(X:(4,3)\), trainable \(W:(3,2)\). Define \(M=XW\), square elementwise, then average all eight values.

\[\texttt{backward()}: \quad L\;\longrightarrow\;Q\;\longrightarrow\;M\;\longrightarrow\;W\]

The forward pass records how the scalar \(L\) depends on \(W\); backward follows that route in reverse and produces \(\nabla_W L:(3,2)\).

The code follows the same forward-then-backward route

import torch
torch.manual_seed(0)
x = torch.randn(4, 3)
w = torch.randn(3, 2, requires_grad=True)
loss = ((x @ w) ** 2).mean()
loss.backward()
print(f"loss={loss.item():.4f}; grad={tuple(w.grad.shape)}")
  

loss=0.5846; grad=(3, 2)

w.grad.shape == w.shape: one gradient value for every trainable value.

Storage and autograd history are separate choices

Start with: graph-connected tensor x. Ask two questions: “Does the result share its memory?” and “Does backward still reach x?”

For a stable snapshot used only for logging or analysis, detach().clone() breaks both connections.

A model needs a reliable inventory of its parameters

Registration is bookkeeping with consequences: the optimizer, device transfer, and checkpoint all use the same inventory.

__init__ registers structure; forward uses it

import torch
from torch import nn
class TinyMLP(nn.Module):
    def __init__(self, d_in, d_hidden, d_out):
        super().__init__()
        self.fc1 = nn.Linear(d_in, d_hidden)
        self.fc2 = nn.Linear(d_hidden, d_out)
    def forward(self, x):
        h = torch.relu(self.fc1(x))
        return self.fc2(h)
  

Define forward(), but call model(x) so module hooks and framework bookkeeping run.

nn.Linear stores the transpose of our S05 convention

For TinyMLP(3,16,2):

\[X:(N,3)\quad\xrightarrow{\;\text{fc1.weight}^{\top}:(3,16)\;}\quad H:(N,16)\quad\xrightarrow{\;\text{fc2.weight}^{\top}:(16,2)\;}\quad Z:(N,2)\]

The mathematical map is unchanged. Only the stored orientation differs.

The registered tree predicts every printed name and shape

model = TinyMLP(3, 16, 2): input width 3, hidden width 16, output width 2.

Registration fixes each parameter's name and shape before any code prints them.

named_parameters prints the predicted inventory

Model as on the previous page: model = TinyMLP(3, 16, 2).

for name, p in model.named_parameters():
    print(name, tuple(p.shape))
  

fc1.weight (16, 3)
fc1.bias   (16,)
fc2.weight (2, 16)
fc2.bias   (2,)
  

The optimizer receives these same four tensors from model.parameters().

The model output and loss must agree on a contract

Pass raw logits to CrossEntropyLoss. It applies log-softmax internally; adding Softmax first changes the objective.

Dataset selects examples; DataLoader forms batches

• TensorDataset adapts tensors already held in memory.
• For evaluation, use a stable order; metrics do not require shuffling.
• Worker processes help when __getitem__ performs real I/O or augmentation, not for every small in-memory dataset.

A Dataset returns one example at a time

from torch.utils.data import TensorDataset

X = torch.randn(10, 3)                        # 10 examples, 3 features
y = torch.tensor([0, 1] * 5, dtype=torch.long)
ds = TensorDataset(X, y)
print(len(ds), ds[0][0].shape, ds[0][1])
  

10 torch.Size([3]) tensor(0)

shuffle=True for training (SGD assumes a fresh random order each epoch); shuffle=False for evaluation (stable and reproducible).

The loader exposes the shapes consumed by the model

import torch
from torch.utils.data import TensorDataset, DataLoader

X = torch.randn(10, 3)
y = torch.tensor([0, 1] * 5, dtype=torch.long)
ds = TensorDataset(X, y)
loader = DataLoader(ds, batch_size=4, shuffle=False)
xb, yb = next(iter(loader))
print(xb.shape, yb.shape, yb.dtype, len(loader))
  

torch.Size([4, 3]) torch.Size([4]) torch.int64 3

The last batch may be smaller. Write the model and loss in terms of \(B\), not a fixed batch size.

Before the loop, decide which object owns each state

Create the model and optimizer outside the loop. Recreating either one discards learned state.

The four-line core changes parameters once per minibatch

Created once before this loop: model, opt, loss_fn, and train_loader. The loader yields xb:(B,D), yb:(B,).

for epoch in range(epochs):
    model.train()
    for xb, yb in train_loader:
        opt.zero_grad()
        logits = model(xb)
        loss = loss_fn(logits, yb)
        loss.backward()
        opt.step()
  

One pass of the inner body is one parameter update: clear, forward, backward, step.

Each line consumes what the previous line produced

The inner body from the previous page: opt.zero_grad() · logits = model(xb) · loss = loss_fn(logits, yb) · loss.backward() · opt.step()

The order is part of the algorithm. step() must use gradients from the current forward pass.

Trace one weight through one iteration

Setup: import torch · import torch.nn as nn · torch.manual_seed(0). A single-weight model, SGD with \(\eta=0.1\).

net = nn.Linear(1, 1, bias=False)     # one scalar weight
opt = torch.optim.SGD(net.parameters(), lr=0.1)
x, y = torch.tensor([[2.0]]), torch.tensor([[1.0]])
w = net.weight
opt.zero_grad();  print(f"zero_grad w={w.item():+.4f}  grad={w.grad}")
loss = nn.MSELoss()(net(x), y)
print(f"forward   w={w.item():+.4f}  grad={w.grad}")
loss.backward();  print(f"backward  w={w.item():+.4f}  grad={w.grad.item():+.4f}")
opt.step();       print(f"step      w={w.item():+.4f}  grad={w.grad.item():+.4f}")
  

zero_grad w=-0.0075  grad=None
forward   w=-0.0075  grad=None
backward  w=-0.0075  grad=-4.0599
step      w=+0.3985  grad=-4.0599
  

\(w_{\text{new}}=w-\eta\,g=-0.0075-0.1\times(-4.0599)=+0.3985\). The weight changes only at step(); the gradient stays until zero_grad().

Evaluation needs two independent switches

eval() and no_grad() solve different problems.

A dataset mean must weight each batch by its size

CrossEntropyLoss returns the mean loss inside each batch. Suppose three batches have sizes \(4,4,2\) and mean losses \(0.4,0.2,0.8\).

Accumulate batch_mean * batch_size, accumulate the example count, then divide once.

Evaluation reduces batch results into dataset metrics

def evaluate(model, loader, loss_fn):
    model.eval()
    total_loss = correct = n = 0
    with torch.no_grad():
        for xb, yb in loader:
            logits = model(xb)
            total_loss += loss_fn(logits, yb).item() * len(xb)
            correct += (logits.argmax(1) == yb).sum().item()
            n += len(xb)
    return total_loss / n, correct / n
  

• The loss function returns a batch mean; multiplying by len(xb) recovers that batch's total.
• Divide once by the total example count, including a smaller last batch.
• The next call to the training loop must restore model.train().

The experiment asks for a nonlinear decision boundary

The data makes the model choice visible: classification requires a nonlinear boundary, while the training loop itself stays unchanged.

Now every component enters one experiment

The program is not a list of APIs: every object fills one slot in the same training system.

Read the result as evidence from one fixed run

Fixed setup: 640 train / 160 validation · 2×64 ReLU MLP · dropout 0.1 · AdamW · seed 0.

epoch  30: train loss 0.2941 | val loss 0.2381 | val acc 0.956
epoch  60: train loss 0.1175 | val loss 0.0597 | val acc 1.000
epoch  90: train loss 0.0586 | val loss 0.0265 | val acc 1.000
epoch 120: train loss 0.0549 | val loss 0.0175 | val acc 1.000
epoch 150: train loss 0.0502 | val loss 0.0120 | val acc 1.000
  

• For this seed and split, all 160 validation examples are classified correctly from epoch 60 onward.

One synthetic run verifies the pipeline end to end. The full experiment is in the s08_pytorch_in_practice notebook.

What one fixed run does not establish

The log from the previous page (640/160 split · 2×64 ReLU MLP · dropout 0.1 · AdamW · seed 0):

epoch  30: train loss 0.2941 | val loss 0.2381 | val acc 0.956
epoch  60: train loss 0.1175 | val loss 0.0597 | val acc 1.000
epoch  90: train loss 0.0586 | val loss 0.0265 | val acc 1.000
epoch 120: train loss 0.0549 | val loss 0.0175 | val acc 1.000
epoch 150: train loss 0.0502 | val loss 0.0120 | val acc 1.000
  

• Accuracy is already saturated while validation loss continues to fall: confidence can change without changing predicted classes.
• Training loss is measured with dropout active; validation metrics use dropout off, so the two losses are not computed under identical model behavior.

The claim stops at this seed and this split. No general optimizer or architecture ranking follows.

Logging should not keep old computation graphs alive

A Python reference to a graph-connected tensor can keep the entire graph reachable and make memory grow across iterations.

Choose checkpoint contents from the task you must resume

# Inference only                      # Resume training
torch.save(model.state_dict(),        torch.save({"model": model.state_dict(),
            "spiralnet.pt")                       "opt":   opt.state_dict(),
                                                  "epoch": epoch}, "resume.pt")
  

A checkpoint is state, not architecture code. Recreate the model class, then load the named tensors.

Loading is a state transfer, not model construction

loaded = SpiralNet()                                  # fresh random weights
state  = torch.load("spiralnet.pt", weights_only=True)
loaded.load_state_dict(state)                         # state transfer
  

A successful load checks structure; a held-out metric checks that the restored system behaves correctly.

Load into a fresh model, then verify behavior

Available from the experiment: trained model, class SpiralNet, evaluate, val_loader, and loss_fn.

import torch
torch.save(model.state_dict(), "spiralnet.pt")
loaded = SpiralNet()
state = torch.load("spiralnet.pt", weights_only=True)
loaded.load_state_dict(state)
loaded.eval()
_, acc_loaded = evaluate(loaded, val_loader, loss_fn)
print(f"loaded validation accuracy: {acc_loaded:.3f}")
  

loaded validation accuracy: 1.000

Loading discipline: strict keys, verified behavior

From the previous page: loaded = SpiralNet() filled via load_state_dict, then evaluated to accuracy 1.000.

• load_state_dict is strict by default: missing or unexpected keys raise immediately.
• Verification checks behavior on held-out data, not just whether loading returned without an exception.

# What strict=True catches (do not silence with strict=False):
Error(s) in loading state_dict for TinyMLP:
    Missing key(s) in state_dict: "fc2.bias".
    Unexpected key(s) in state_dict: "typo.weight".
  

A checkpoint that loads without an exception has not yet been shown to work.

A silent shape mismatch trains on the wrong loss

Setup: import torch · import torch.nn as nn · torch.manual_seed(0). The model emits pred:(8,); the loader column arrives as target:(8, 1).

pred = torch.randn(8)
target = torch.randn(8, 1)
wrong = nn.MSELoss()(pred, target)            # runs without error
right = nn.MSELoss()(pred, target.squeeze(1))
print((pred - target).shape)
print(f"wrong: {wrong.item():.4f}   right: {right.item():.4f}")
  

torch.Size([8, 8])
wrong: 1.8507   right: 2.0946
  

Subtraction broadcasts \((8,)\) against \((8,1)\) to \((8,8)\): the mean runs over 64 pairwise differences instead of 8 matched ones.

Why nothing failed, and the one-line defense

From the previous page: pred:(8,) vs target:(8, 1) gave loss 1.8507 where the matched loss is 2.0946.

• Broadcasting is a defined operation, so no rule was violated: the program computes a valid mean over the wrong \((8,8)\) tensor.
• PyTorch emits a UserWarning naming the two sizes, but warnings scroll past in a training log, and gradient descent optimizes the wrong number without complaint.
• The number is differently wrong every batch, so the loss curve can still fall while measuring the wrong thing.

assert pred.shape == target.shape next to the loss turns this silent bug into a loud one.

Use the symptom to choose the first object to inspect

Check the system, not isolated API names

A model's validation result changes after loading. Name three different contracts you would verify.

Model architecture and keys; preprocessing / label mapping; evaluation mode and the held-out dataset.

Six conventions from today

These six rules recur in every PyTorch program this course builds.

For next time

• Read: 2.1 Convolution.
• Quiz 1 in S09: the last 30 minutes (6:45–7:15 PM), on Canvas, covering S01–S08.
• Then: keep this training loop and replace the MLP with a model that uses image structure.