Session S09
Why images call for local connections and shared detectors.
Deep Learning · CS 4715/5715 · University of Idaho
Quiz 1 in Canvas at the end of class, 6:45–7:15 PM.
(N,C,H,W)loss.backward()optimizer.step()An image batch has shape \((N,C,H,W)\): \(N\) images, \(C\) channels (3 for RGB), height \(H\), width \(W\). The loop still expects logits of shape (\(N\), number of classes).
Today's question: what structure of an image should the model keep?
The DataLoader delivers batches of shape \((N,C,H,W)\). nn.Linear needs one vector per example.
import torch
from torch import nn
x = torch.randn(8, 3, 32, 32) # 8 RGB images, 32x32
flat = nn.Flatten()(x) # keep axis 0, join the rest
print(flat.shape)
torch.Size([8, 3072])\(3\cdot32\cdot32=3072\) numbers per image, and a dense layer gives every one of them its own weights.
one \(4\times4\) channel, pixels numbered row by row
A dense layer has no built-in order: shuffle the pixels and its weight columns the same way, and the outputs do not change.
Nothing in a dense layer marks which inputs were neighbors. A convolutional layer keeps the grid.
A layer that keeps the grid can compute each output from one neighborhood. The same vertical edge appears at two positions:
red line = vertical edge
between columns 1 and 2 (counting from 0)
red line = vertical edge
between columns 2 and 3
One detector reused at every location: what it learns at one position applies at all positions.
One output reads one neighborhood.
Every location uses the same weights.
The shared weight array is a kernel (also called a filter). Its responses over the grid form a feature map.
Compute one kernel response by hand: a \(4\times4\) input \(X\) and a \(2\times2\) kernel \(K\).
\(X\): rows 0–1, columns 0–1
| 1 | 2 | 0 | 1 |
| 3 | 1 | 2 | 2 |
| 0 | 1 | 3 | 1 |
| 2 | 2 | 1 | 0 |
patch
| 1 | 2 |
| 3 | 1 |
\(K\)
| 1 | 0 |
| −1 | 1 |
\(Y_{0,0}\)
\(1(1)+2(0)+3(-1)+1(1)=-1\)
\(\odot\): multiply matching entries. Adding the four products turns the \(2\times2\) patch into one number.
Repeat at the next window starts (start = top-left cell of the window), moving one column at a time: \(K=\begin{bmatrix}1&0\\-1&1\end{bmatrix}\).
start \((0,0)\) → \(Y_{0,0}=-1\)
| 1 | 2 | 0 | 1 |
| 3 | 1 | 2 | 2 |
| 0 | 1 | 3 | 1 |
| 2 | 2 | 1 | 0 |
start \((0,1)\) → \(Y_{0,1}=3\)
| 1 | 2 | 0 | 1 |
| 3 | 1 | 2 | 2 |
| 0 | 1 | 3 | 1 |
| 2 | 2 | 1 | 0 |
start \((0,2)\) → \(Y_{0,2}=0\)
| 1 | 2 | 0 | 1 |
| 3 | 1 | 2 | 2 |
| 0 | 1 | 3 | 1 |
| 2 | 2 | 1 | 0 |
The patch changes at each start. The four values in \(K\) do not.
Every start that fits gives one entry: 3 starts per row and 3 per column.
\(X:4\times4\)
| 1 | 2 | 0 | 1 |
| 3 | 1 | 2 | 2 |
| 0 | 1 | 3 | 1 |
| 2 | 2 | 1 | 0 |
apply \(K\) at every start
| 1 | 0 |
| −1 | 1 |
\(Y:3\times3\)
| −1 | 3 | 0 |
| 4 | 3 | 0 |
| 0 | 0 | 2 |
Each entry of \(Y\) is the inner product of \(K\) with the patch at that location, largest when the patch, read as a vector, points in the same direction as \(K\). Its position in \(Y\) records where.
Libraries call this operation convolution. Strictly it is cross-correlation: \(K\) is used as stored, while true convolution first flips \(K\) in both directions (a 180° rotation). A learned kernel simply learns the flipped weights.
Scope: one image and channel, one kernel \(K\in\mathbb{R}^{k_h\times k_w}\), no bias, window moving one cell at a time, no zeros added around the image.
\[Y_{i,j}=\sum_{u=0}^{k_h-1}\sum_{v=0}^{k_w-1}X_{i+u,j+v}\,K_{u,v}\]
| Symbol | Role | Changes while the kernel moves? |
|---|---|---|
| \(i,j\) | Output location = patch start | Yes |
| \(u,v\) | Position inside the patch and the kernel | Summed over |
| \(K_{u,v}\) | One learned weight inside \(K\) | No, it is shared |
This is the patch-times-kernel sum computed by hand, written once for every output location \((i,j)\).
So far every window start moved by one cell. Here: \(3\times3\) kernel, stride \(S=2\), no zeros added around the image (cells numbered 1–25 only to name them).
start \((0,0)\)
| 1 | 2 | 3 | 4 | 5 |
| 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 |
| 21 | 22 | 23 | 24 | 25 |
next start \((0,2)\)
| 1 | 2 | 3 | 4 | 5 |
| 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 |
| 21 | 22 | 23 | 24 | 25 |
Starts 0 and 2 fit along each axis, so this \(5\times5\) input gives a \(2\times2\) output.
A larger stride never evaluates more windows, and usually fewer, so the output never grows.
Without padding, a \(3\times3\) window can never be centered on a border pixel.
original \(3\times3\) image
| 1 | 2 | 3 |
| 4 | 5 | 6 |
| 7 | 8 | 9 |
padded \(5\times5\) input; first \(3\times3\) window
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 2 | 3 | 0 |
| 0 | 4 | 5 | 6 | 0 |
| 0 | 7 | 8 | 9 | 0 |
| 0 | 0 | 0 | 0 | 0 |
The first window is now centered on pixel 1. With \(S=1\), the padded \(5\times5\) input fits 3 starts per axis, so the output is \(3\times3\): the original size.
Stride changes the distance between starts. Padding changes the input the windows can see.
One axis: \(I=8\) inputs, padding \(P=1\), kernel size \(k=3\), stride \(S=2\). Window \(m\) covers padded cells \(mS\) to \(mS+k-1\); the last cell is \(I+2P-1=9\).
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| 0 | \(x_0\) | \(x_1\) | \(x_2\) | \(x_3\) | \(x_4\) | \(x_5\) | \(x_6\) | \(x_7\) | 0 |
blue: window \(m=0\), start 0 · green: window \(m=3\), start 6, the last that fits (start 8 would need cells 8–10)
\[mS+k\le I+2P\quad\Rightarrow\quad m_{\max}=\left\lfloor\frac{I+2P-k}{S}\right\rfloor\]
\[\boxed{O=m_{\max}+1=\left\lfloor\frac{I+2P-k}{S}\right\rfloor+1}\]
Here \(O=\lfloor 7/2\rfloor+1=4\): starts 0, 2, 4, 6. The \(+1\) counts window \(m=0\); \(\lfloor\cdot\rfloor\) drops a window that does not fully fit.
Apply the formula to \(I=8\) and \(k=3\) with three settings.
| Padding \(P\) | Stride \(S\) | Calculation | Output \(O\) |
|---|---|---|---|
| 0 | 1 | \(\lfloor(8-3)/1\rfloor+1\) | 6 |
| 1 | 1 | \(\lfloor(8+2-3)/1\rfloor+1\) | 8 |
| 1 | 2 | \(\lfloor(8+2-3)/2\rfloor+1\) | 4 |
Predict: for an odd \(k\), what padding keeps the size when \(S=1\)?
Answer: \(P=(k-1)/2\), for example \(P=1\) for \(k=3\). With \(S=1\), \(O=I+2P-k+1\), which equals \(I\) when \(2P=k-1\); an odd \(k\) makes \(P\) a whole number.
Scope: single image, single input channel, single kernel, no bias.
import numpy as np
def conv2d_np(x, k, stride=1, padding=0): # k: the kernel array K
x = np.pad(x, padding) # add `padding` zeros on every side
kh, kw = k.shape
oh = (x.shape[0] - kh) // stride + 1 # x is padded: x.shape[0] = I + 2P; // rounds down
ow = (x.shape[1] - kw) // stride + 1
out = np.empty((oh, ow), dtype=np.float32) # every entry is filled below
for i in range(oh):
for j in range(ow):
r, c = i * stride, j * stride # window start (iS, jS)
patch = x[r:r + kh, c:c + kw] # kh x kw slice of the input
out[i, j] = np.sum(patch * k) # multiply matching entries, then add
return out
Output index \((i,j)\) picks the start \((iS,jS)\); the slice is the patch; np.sum(patch * k) is the double sum.
Run conv2d_np from the previous slide on the hand-computed \(X\) and \(K\).
X = np.array([[1, 2, 0, 1],
[3, 1, 2, 2],
[0, 1, 3, 1],
[2, 2, 1, 0]], dtype=np.float32)
K = np.array([[1, 0],
[-1, 1]], dtype=np.float32)
print(conv2d_np(X, K)) # stride 1, padding 0
[[-1. 3. 0.]
[ 4. 3. 0.]
[ 0. 0. 2.]]
by hand, \(Y:3\times3\)
| −1 | 3 | 0 |
| 4 | 3 | 0 |
| 0 | 0 | 2 |
first entry: \(1(1)+2(0)+3(-1)+1(1)=-1\)
A small hand-computed case is the first test for any convolution code: every entry can be checked.
F.conv2d on random inputs
import torch
import torch.nn.functional as F # functional API: you pass the weight tensor yourself
# uses np and conv2d_np from the loop slide
np.random.seed(0) # fixed seed: same numbers every run
x = np.random.randn(8, 8).astype(np.float32)
k = np.random.randn(3, 3).astype(np.float32) # asymmetric: differs from its 180-degree rotation
for stride, padding in [(1, 0), (1, 1), (2, 1)]:
ours = conv2d_np(x, k, stride, padding)
xt = torch.from_numpy(x)[None, None] # array -> tensor; each None adds an axis of size 1
kt = torch.from_numpy(k)[None, None] # xt: (1, 1, 8, 8) = (N, C, H, W); kt: (1, 1, 3, 3)
ref = F.conv2d(xt, kt, stride=stride, padding=padding)[0, 0].numpy() # drop N, C; to NumPy
print(ours.shape, np.max(np.abs(ours - ref)))
(6, 6) 7.1525574e-07
(8, 8) 9.536743e-07
(4, 4) 4.7683716e-07
Shapes match the three formula cases; differences near \(10^{-7}\) are float32 rounding. Because \(k\) is asymmetric, the match also shows that F.conv2d uses \(K\) as stored, without the 180° rotation.
With the code verified, choose \(K\) by hand to detect a dark-to-bright boundary (0 = dark, 1 = bright).
input image \(6\times8\); orange: the window over columns 2–4
vertical-edge kernel
| −1 | 0 | 1 |
| −1 | 0 | 1 |
| −1 | 0 | 1 |
\(4\times6\) feature map
| 0 | 0 | 3 | 3 | 0 | 0 |
| 0 | 0 | 3 | 3 | 0 | 0 |
| 0 | 0 | 3 | 3 | 0 | 0 |
| 0 | 0 | 3 | 3 | 0 | 0 |
The window over columns 2–4 reads \([0\;0\;1]\) in every row: \(-1\cdot0+0\cdot0+1\cdot1=1\), and three rows give 3. The window over columns 3–5 reads \([0\;1\;1]\) and also gives 3, so two output columns respond. Flat windows give 0.
Here \(K\) is chosen by hand to show what a kernel measures. In a trained CNN, \(K\) is learned by gradient descent like any other weight.
import torch
import torch.nn.functional as F
img = torch.zeros(1, 1, 6, 8) # (N, C, H, W): one 6x8 image, all 0 (dark)
img[:, :, :, 4:] = 1.0 # columns 4 to 7 become 1 (bright)
img_shift = torch.zeros(1, 1, 6, 8)
img_shift[:, :, :, 5:] = 1.0 # boundary one column further right
edge_k = torch.tensor([[-1.0, 0.0, 1.0],
[-1.0, 0.0, 1.0],
[-1.0, 0.0, 1.0]]).view(1, 1, 3, 3) # shape (1, 1, 3, 3), like [None, None]
edges = F.conv2d(img, edge_k) # stride 1, no padding: 6x8 -> 4x6
edges_shift = F.conv2d(img_shift, edge_k)
print(edges[0, 0, 0].tolist()) # example 0, channel 0, row 0, as a Python list
print(edges_shift[0, 0, 0].tolist())
[0.0, 0.0, 3.0, 3.0, 0.0, 0.0]
[0.0, 0.0, 0.0, 3.0, 3.0, 0.0]
Every patch moves one column, and \(K_{u,v}\) does not depend on \((i,j)\): translation equivariance, away from the border.
It needs stride 1: with stride 2, a one-column shift has no matching output shift.
Neighboring cells of the edge map repeat the same response. A \(2\times2\) max pool keeps one value per non-overlapping window.
edge feature map: \(4\times6\)
pooled map: \(2\times3\)
| 0 | 3 | 0 |
| 0 | 3 | 0 |
Later layers now process 6 locations instead of 24. The exact position of a response inside each window is discarded.
Pooling trades spatial detail for a smaller map.
Pool both edge maps (F, edges, edges_shift from the shifted-image code). Pooling has no parameters and works on each channel separately.
pooled = F.max_pool2d(edges, kernel_size=2, stride=2) # 2x2 window, no weights: 4x6 -> 2x3
pooled_shift = F.max_pool2d(edges_shift, kernel_size=2, stride=2)
print(pooled.shape)
print(pooled[0, 0, 0].tolist())
print(pooled_shift[0, 0, 0].tolist())
torch.Size([1, 1, 2, 3])
[0.0, 3.0, 0.0]
[0.0, 3.0, 3.0]
The middle window still holds a 3, so its maximum stays 3. The third window gained a 3, so its value changed from 0 to 3.
A \(3\times3\) kernel sees a \(3\times3\) region, but an object can fill much of the image. Networks stack conv layers, usually with a ReLU after each. Which input pixels can one second-layer activation depend on?
When every layer has stride 1, each \(3\times3\) layer adds one input cell on each side: 1 → 3 → 5 → 7. In 2-D the region is \(3\times3\), then \(5\times5\), then \(7\times7\). A ReLU does not change which inputs can matter.
The receptive field of an activation is the input region that can affect it. Depth makes it grow.
Downsampling (stride 2 or \(2\times2\) pooling) keeps every second position, so neighboring activations sit 2 input pixels apart. Call that distance the jump \(j\) (a distance, not a column index). A \(3\times3\) layer after it:
\[ j_\ell=j_{\ell-1}S_\ell,\qquad r_\ell=r_{\ell-1}+(k_\ell-1)\,j_{\ell-1},\qquad r_0=j_0=1 \]
\(r_\ell\): receptive field after layer \(\ell\); \(k_\ell,S_\ell\): that layer's kernel size and stride. A \(2\times2\) pool counts as a layer with \(k=2,\ S=2\): it adds \((2-1)j\) to \(r\) and doubles \(j\).
In words: each layer multiplies the jump by its stride and widens the field by \(k-1\) jumps. With all \(3\times3\), stride-1 layers: \(r=1+2L\).
Run the two recurrences on three \(3\times3\) conv layers, with and without a stride-2 middle layer.
| Layer | \(j\) | \(r\) |
|---|---|---|
| Input | 1 | 1 |
| Conv 1 | 1 | 3 |
| Conv 2 | 1 | 5 |
| Conv 3 | 1 | 7 |
| Layer | \(j\) | \(r\) |
|---|---|---|
| Input | 1 | 1 |
| Conv 1 | 1 | 3 |
| Conv 2, \(S=2\) | 2 | 5 |
| Conv 3 | 2 | 9 |
Conv 3 adds \((3-1)\cdot1=2\) in the all-stride-1 network, but \((3-1)\cdot2=4\) in the stride-2 network.
Stride 2 changes the spacing first; the next kernel then covers a wider part of the input.
Every example so far had one input channel. A color image has three: red, green, and blue.
\(C_{\text{out}}\) feature maps
A kernel has one \(k_h\times k_w\) slice per input channel; the products from all slices are summed into one number per location.
A layer with \(C_{\text{out}}\) kernels produces \(C_{\text{out}}\) feature maps: the output channels.
The single-channel rule, extended with a batch index, channels, stride, padding, and bias.
Input \(X:(N,C_{\text{in}},H,W)\), kernels \(K:(C_{\text{out}},C_{\text{in}},k_h,k_w)\), bias \(b:(C_{\text{out}})\), \(\widetilde X\) = \(X\) padded with zeros.
\[ Y_{n,o,i,j}=b_o+ \sum_{c=0}^{C_{\text{in}}-1} \sum_{u=0}^{k_h-1}\sum_{v=0}^{k_w-1} K_{o,c,u,v}\,\widetilde X_{n,c,iS+u,jS+v} \]
\(n,o,i,j\): one example, one output channel, one location.
\(iS+u,\ jS+v\) run over the padded window.
Sum over \(c,u,v\), then add the bias \(b_o\).
Output shape: \((N,C_{\text{out}},O_H,O_W)\), with each \(O\) from the output-size formula.
nn.Conv2d stores those tensors as a module
import torch
from torch import nn
conv = nn.Conv2d(3, 16, kernel_size=3, padding=1) # C_in=3, C_out=16, 3x3 kernel, P=1
x = torch.randn(8, 3, 32, 32) # (N, C_in, H, W)
y = conv(x) # stride 1 by default
print(conv.weight.shape, conv.bias.shape, y.shape)
print(sum(p.numel() for p in conv.parameters())) # all weights plus all biases
torch.Size([16, 3, 3, 3]) torch.Size([16]) torch.Size([8, 16, 32, 32])
448
\(C_{\text{out}}C_{\text{in}}k_hk_w+C_{\text{out}}=16\cdot3\cdot3\cdot3+16=448\). A dense layer from \(3\cdot32\cdot32=3072\) inputs to \(16\cdot32\cdot32=16384\) outputs needs \(3072\cdot16384+16384=50{,}348{,}032\).
conv.weight holds \(K\), shape \((C_{\text{out}},C_{\text{in}},k_h,k_w)\). The count depends on channels and kernel size, not on the image size.
A \(3\times3\) kernel mixes channels and neighboring pixels. With \(k_h=k_w=1\), only the channel mixing remains: \(\;Y_{n,o,i,j}=b_o+\sum_c K_{o,c,0,0}\,X_{n,c,i,j}\)
# torch and nn imported on the nn.Conv2d slide
mix = nn.Conv2d(64, 16, kernel_size=1) # 64 channels in, 16 out, 1x1 kernel
y = mix(torch.randn(8, 64, 32, 32))
print(mix.weight.shape, y.shape)
print(sum(p.numel() for p in mix.parameters()))
torch.Size([16, 64, 1, 1]) torch.Size([8, 16, 32, 32])
1040
mix.weight[:, :, 0, 0] is a \(16\times64\) matrix: the same map as nn.Linear(64, 16), applied to the channel vector of every pixel.
Spatial size stays \(32\times32\). Cost: \(64\cdot16+16=1040\) parameters.
Put the pieces together as the model in the S08 training loop (torch and nn as imported before). Comments give each layer's output shape.
model = nn.Sequential( # applies the layers in order
nn.Conv2d(3, 16, kernel_size=3, padding=1), # (8, 16, 32, 32)
nn.ReLU(),
nn.MaxPool2d(2), # 2x2 windows, stride 2: (8, 16, 16, 16)
nn.Conv2d(16, 32, kernel_size=3, padding=1), # (8, 32, 16, 16)
nn.ReLU(),
nn.MaxPool2d(2), # (8, 32, 8, 8)
nn.Flatten(), # (8, 2048)
nn.Linear(32 * 8 * 8, 10), # (8, 10): logits for 10 classes
)
x = torch.randn(8, 3, 32, 32)
print(model(x).shape, sum(p.numel() for p in model.parameters()))
torch.Size([8, 10]) 25578This answers today's question: the model keeps the grid, with local windows and shared kernels, and flattens only at the end. The training loop does not change.
An input has shape \((8,3,32,32)\). Apply nn.Conv2d(3, 8, kernel_size=5, stride=2, padding=2). What are the output shape and the parameter count? Why are there fewer parameters than in a dense image-to-image map?
\(O=\lfloor(32+2\cdot2-5)/2\rfloor+1=\lfloor15.5\rfloor+1=16\), so the output \((N,C_{\text{out}},O_H,O_W)\) is \((8,8,16,16)\).
Parameters: \(8\cdot3\cdot5\cdot5+8=608\). Stride changes the output shape, not the parameter count.
Each output reads a local \(5\times5\) window (locality), and the same 608 parameters are shared by all positions (weight sharing).
Everyone completes Quiz 1 in Canvas. Covers Module 1: S01–S08.
Online (Web) students and students with flexible-time arrangements may use their assigned flexible completion window.
Before S10: finish Reading 2.1 (Convolution) and read 2.2 (From LeNet to ResNet). S10 stacks these blocks into deeper networks inside the same training loop.