Module 2 · Reading 2.1 · read before S09: CNNs I — Convolution

Take the MLPs from Section 1.2 and point them at images, and two things go wrong at once. First, the parameter count explodes: a single fully connected layer from a 32x32 RGB image to a same-sized feature map needs about fifty million weights, and that is a small image. Second, and more fundamentally, a dense layer has no idea that pixels form a grid. It treats pixel (0,0) and pixel (31,31) as unrelated coordinates, so a cat detector learned in the top-left corner tells the network nothing about cats in the bottom-right. Every position must relearn the same pattern from scratch.

Convolution fixes both problems with one idea: slide a small window of weights across the image and reuse it everywhere. This section builds the operation from the ground up (kernels, stride, padding, pooling, and receptive fields) and verifies our own NumPy implementation against PyTorch’s, digit for digit. Everything in Sections 2.2 and 2.3 is built out of the pieces defined here.

How to read this

From dense layers to local, shared weights

A convolutional layer makes two structural bets about images, and both are forms of prior knowledge baked into the architecture.

Locality. Meaningful low-level structure (an edge, a corner, a patch of texture) is local. To decide whether there is a vertical edge at some position, you need the few pixels around that position, not the whole image. So instead of connecting an output unit to every input pixel, connect it to a small neighborhood, say 3x3.

Translation structure (parameter sharing). An edge detector that works at one position works at every position. So use the same 3x3 weights at every location, rather than learning a separate detector per location. The set of shared weights is called a kernel (or filter), and sliding it across the input produces a feature map: a grid of responses, one per position.

The savings are not subtle. A 3x3 kernel mapping 3 input channels to 16 output channels costs 16 * 3 * 3 * 3 + 16 = 448 parameters, no matter how large the image is. The fully connected layer computing an output of the same shape from a 32x32 image costs over 50 million (we count both below, in running code). Fewer parameters means less overfitting and less memory, but the deeper win is statistical: every training image teaches the kernel about every position at once. In the language of Chapter 1, convolution is a massive, hard-wired form of regularization — a constraint on the hypothesis space that happens to match how images work.

Historical note. The design is older than backpropagation-trained networks. In experiments published around 1959–1962, David Hubel and Torsten Wiesel recorded from neurons in cat visual cortex and found “simple cells” that fire for oriented edges at specific positions, feeding “complex cells” that respond to the same orientation with some position tolerance — work that later earned a Nobel Prize. Kunihiko Fukushima’s Neocognitron (1980) turned that hierarchy into an artificial network with alternating feature-detecting and pooling layers, but lacked an effective way to learn the weights from data. Yann LeCun and colleagues supplied that in 1989 by training a convolutional network with backpropagation to read handwritten ZIP code digits, the direct ancestor of every CNN in this chapter.

The convolution operation, precisely

Start with a single-channel input x of shape (H, W) and a kernel k of shape (kh, kw). The output at position (i, j) is the dot product between the kernel and the input patch it currently covers:

\[\text{out}[i,\, j] = \sum_a \sum_b \, x[i+a,\; j+b] \cdot k[a,\, b]\]

with a ranging over 0..kh-1 and b over 0..kw-1. That is the whole operation: an inner product, repeated at every position. Two knobs control how the window moves:

  • Stride is the step size between window positions. Stride 1 visits every position; stride 2 skips every other one and usually produces about half as many positions. The output-size formula below gives the exact result.
  • Padding adds a border of zeros around the input before sliding. Without padding, a 3x3 kernel cannot center itself on border pixels, so each conv layer shrinks the map by 2 pixels per dimension; padding by 1 preserves the size (“same” padding for 3x3).

The output size along each dimension is:

\[\text{out} = \lfloor (\text{in} + 2 \cdot \text{padding} - \text{kernel}) / \text{stride} \rfloor + 1\]

Here is a from-scratch NumPy implementation checked against torch.nn.functional.conv2d for three stride/padding settings:

import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
torch.manual_seed(0)

def conv2d_np(x, k, stride=1, padding=0):
    """2D 'convolution' as deep learning frameworks define it (cross-correlation).
    x: (H, W) input, k: (kh, kw) kernel."""
    if padding > 0:
        x = np.pad(x, padding)
    kh, kw = k.shape
    H, W = x.shape
    out_h = (H - kh) // stride + 1
    out_w = (W - kw) // stride + 1
    out = np.zeros((out_h, out_w), dtype=np.float32)
    for i in range(out_h):
        for j in range(out_w):
            patch = x[i * stride : i * stride + kh, j * stride : j * stride + kw]
            out[i, j] = np.sum(patch * k)
    return out

x = np.random.randn(8, 8).astype(np.float32)
k = np.random.randn(3, 3).astype(np.float32)

for stride, padding in [(1, 0), (1, 1), (2, 1)]:
    ours = conv2d_np(x, k, stride=stride, padding=padding)
    torch_out = F.conv2d(
        torch.from_numpy(x).view(1, 1, 8, 8),
        torch.from_numpy(k).view(1, 1, 3, 3),
        stride=stride, padding=padding,
    )[0, 0].numpy()
    max_diff = np.abs(ours - torch_out).max()
    print(f"stride={stride} padding={padding}: "
          f"output {ours.shape[0]}x{ours.shape[1]}, "
          f"max |numpy - torch| = {max_diff:.2e}")

Running this prints:

stride=1 padding=0: output 6x6, max |numpy - torch| = 7.15e-07
stride=1 padding=1: output 8x8, max |numpy - torch| = 9.54e-07
stride=2 padding=1: output 4x4, max |numpy - torch| = 4.77e-07

The disagreements are on the order of 1e-07: float32 rounding, not algorithmic difference. Check the shapes against the formula: with an 8x8 input and 3x3 kernel, no padding gives (8 - 3)/1 + 1 = 6; padding 1 gives (8 + 2 - 3)/1 + 1 = 8 (size preserved); stride 2 with padding 1 gives floor(7/2) + 1 = 4. Matching a reference implementation numerically, exactly like this, is how you will validate your own convolution in HW2.

Pitfall. What deep learning frameworks call “convolution” is, in the signal-processing sense, cross-correlation: the kernel is slid as-is, not flipped. True convolution flips the kernel in both spatial dimensions first. For learned kernels the distinction is invisible (the network would learn the flipped weights instead), but it bites when you port hand-designed kernels from image-processing references, or compare against scipy.signal.convolve2d, which does flip. If your outputs look mirrored against a textbook’s, this is why.

Kernels as pattern detectors

What does the inner product actually measure? The response is largest where the input patch points in the same direction as the kernel, viewed as vectors. So a kernel is a template, and its feature map answers, at every position: “how much does this neighborhood look like my pattern?”

You can see this with a hand-built kernel. The classic vertical-edge detector has negative weights in its left column and positive weights in its right column; it responds strongly where dark pixels sit left of bright ones, and produces zero on flat regions:

import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
torch.manual_seed(0)

# An image that is dark (0) on the left half, bright (1) on the right half.
img = torch.zeros(1, 1, 6, 8)
img[:, :, :, 4:] = 1.0

# A hand-designed vertical-edge kernel: responds where left != right.
edge_k = torch.tensor([[-1.0, 0.0, 1.0],
                       [-1.0, 0.0, 1.0],
                       [-1.0, 0.0, 1.0]]).view(1, 1, 3, 3)

edges = F.conv2d(img, edge_k)  # no padding: 6x8 -> 4x6
print("edge response (one row shown; all rows identical):")
print(edges[0, 0, 0].tolist())

# The same kernel applied to the image shifted right by one pixel:
img_shift = torch.zeros(1, 1, 6, 8)
img_shift[:, :, :, 5:] = 1.0
edges_shift = F.conv2d(img_shift, edge_k)
print("edge response on shifted image:")
print(edges_shift[0, 0, 0].tolist())

# Max pooling makes the detection tolerant to that shift.
pooled = F.max_pool2d(edges, kernel_size=2, stride=2)
pooled_shift = F.max_pool2d(edges_shift, kernel_size=2, stride=2)
print(f"after 2x2 max pool:         {pooled[0, 0, 0].tolist()}")
print(f"after 2x2 max pool (shift): {pooled_shift[0, 0, 0].tolist()}")

# Parameter counts: conv layer vs fully connected layer on a 32x32 RGB image.
conv = torch.nn.Conv2d(3, 16, kernel_size=3, padding=1)   # 32x32x3 -> 32x32x16
fc = torch.nn.Linear(32 * 32 * 3, 32 * 32 * 16)           # same input/output sizes
n_conv = sum(p.numel() for p in conv.parameters())
n_fc = sum(p.numel() for p in fc.parameters())
print(f"conv 3->16, 3x3 kernel:  {n_conv:,} parameters")
print(f"fully connected version: {n_fc:,} parameters")
print(f"ratio: {n_fc / n_conv:,.0f}x")

Running this prints:

edge response (one row shown; all rows identical):
[0.0, 0.0, 3.0, 3.0, 0.0, 0.0]
edge response on shifted image:
[0.0, 0.0, 0.0, 3.0, 3.0, 0.0]
after 2x2 max pool:         [0.0, 3.0, 0.0]
after 2x2 max pool (shift): [0.0, 3.0, 3.0]
conv 3->16, 3x3 kernel:  448 parameters
fully connected version: 50,348,032 parameters
ratio: 112,384x

Three observations. The kernel fires (response 3.0) exactly where its window straddles the dark-to-bright boundary and is silent on flat regions — a position-tagged edge detector, built from 9 numbers. When the edge shifts one pixel right, the response pattern shifts one pixel right with it; this property, equivariance, is parameter sharing made visible: shift the input, and the feature map shifts identically. And the parameter count confirms the arithmetic from earlier: 448 versus about 50 million for the dense layer computing the same-shaped output.

Pooling: buying tolerance to position

Equivariance means the information moves with the input. Often you want the opposite at higher levels: “there is an edge around here” rather than “at column 3 exactly.” Pooling provides that. A 2x2 max-pooling layer replaces each non-overlapping 2x2 block of a feature map with its maximum (average pooling uses the mean), halving each spatial dimension and keeping the strongest response in each block.

Look at the pooled rows in the output above. Before pooling, the one-pixel shift moved every response. After pooling, the middle pooled cell reads 3.0 in both versions — within a pooling window, position information is discarded, so the detection survives the shift unchanged. The third cell does change (0.0 to 3.0), because the shifted response crossed a window boundary. Pooling therefore buys tolerance to translations up to roughly the pooling width, not true invariance; stacking conv/pool stages grows that tolerance level by level. This is precisely Hubel and Wiesel’s simple-to-complex cell arrangement, and Fukushima’s S-cell/C-cell alternation, rediscovered as an engineering necessity.

Modern architectures often get their downsampling from stride-2 convolutions instead of explicit pooling layers, and replace the final pooling with a global average pool (average each feature map to a single number), which removes the need for a fixed input size. Either way, some mechanism for discarding precise position is doing essential work.

Receptive fields: how a 3x3 kernel sees the whole image

A single 3x3 kernel sees 9 pixels. How can a network built from such myopic units recognize objects that span hundreds? The answer is depth. The receptive field of a unit is the region of the input that can influence its value. Stack two 3x3 convs and each unit in the second map depends on a 3x3 patch of the first map, whose units each depend on a 3x3 patch of input: 5x5 of input in total. Each additional 3x3 layer adds 2:

\[\text{receptive field}(L \text{ stacked 3x3 convs}) = 1 + 2L\]

Downsampling accelerates this. After a stride-2 layer, every subsequent layer’s kernel steps across the original image twice as fast, so its contribution to the receptive field doubles; after two downsamplings, it quadruples. A stack of 3x3 layers interleaved with a few stride-2 stages reaches image-scale receptive fields within a dozen layers, while every individual layer stays cheap.

The bookkeeping generalizes to any stack, and it is worth doing once by hand and once in code. Track two numbers layer by layer: the jump j, the product of all strides so far (how many input pixels one step in the current feature map corresponds to), and the receptive field rf itself. Each layer with kernel size k grows the receptive field by (k - 1) * j, using the jump before that layer, and then multiplies the jump by its own stride:

rf_new = rf + (k - 1) * j, then j_new = j * s

The script below runs this recursion on a typical small stack (conv, conv, pool, conv, pool, conv) and then checks the answer empirically: it builds the same stack with all-positive weights, backpropagates from a single central output unit, and measures which input pixels received nonzero gradient. That set of pixels is the receptive field, measured rather than derived.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# Receptive-field arithmetic. Track two numbers layer by layer:
#   jump j  = product of all strides so far (input pixels per output step)
#   rf      = receptive field; each layer adds (kernel - 1) * incoming jump
layers = [
    ("conv 3x3 s1", 3, 1),
    ("conv 3x3 s1", 3, 1),
    ("avgpool 2 s2", 2, 2),
    ("conv 3x3 s1", 3, 1),
    ("avgpool 2 s2", 2, 2),
    ("conv 3x3 s1", 3, 1),
]
rf, j = 1, 1
print(f"{'layer':<13} {'k':>2} {'s':>2} {'rf':>4} {'jump':>5}")
for name, k, s in layers:
    rf = rf + (k - 1) * j
    j = j * s
    print(f"{name:<13} {k:>2} {s:>2} {rf:>4} {j:>5}")

# Empirical check: which input pixels can influence one output unit?
# Build the same stack with all-positive weights (so no cancellation),
# backprop from a single central output unit, and measure the support.
convs = []
for name, k, s in layers:
    if name.startswith("conv"):
        c = nn.Conv2d(1, 1, 3, padding=0, bias=False)
        with torch.no_grad():
            c.weight.fill_(0.1)
        convs.append(c)
    else:
        convs.append(nn.AvgPool2d(2))
net = nn.Sequential(*convs)

x = torch.ones(1, 1, 64, 64, requires_grad=True)
out = net(x)
h = out.shape[-1] // 2
out[0, 0, h, h].backward()
support = (x.grad[0, 0] != 0)
rows = support.any(dim=1).nonzero().squeeze(1)
cols = support.any(dim=0).nonzero().squeeze(1)
print(f"output map is {out.shape[-2]}x{out.shape[-1]}; "
      f"one central unit's input support: "
      f"{rows.max() - rows.min() + 1} x {cols.max() - cols.min() + 1} pixels")

Running this prints:

layer          k  s   rf  jump
conv 3x3 s1    3  1    3     1
conv 3x3 s1    3  1    5     1
avgpool 2 s2   2  2    6     2
conv 3x3 s1    3  1   10     2
avgpool 2 s2   2  2   12     4
conv 3x3 s1    3  1   20     4
output map is 12x12; one central unit's input support: 20 x 20 pixels

Read the table against the formula. The first two conv layers give \(\text{rf} = 1 + 2L\): 3, then 5. The first pool adds only 1 (its kernel is 2) but doubles the jump, so the next conv adds 4 instead of 2 (receptive field 10). After the second pool the jump is 4, and the final conv adds 8, landing at 20. Four weight layers, receptive field 20 pixels; the same four convs without the pools would reach only 9. And the gradient measurement agrees exactly: a 20x20 patch of the input can influence the chosen output unit. (The check uses average pooling because max pooling routes gradients only through each window’s argmax, which would understate the set of pixels that could matter.)

One refinement worth knowing: the theoretical receptive field says which pixels can influence a unit, not how much they do. Luo, Li, Urtasun, and Zemel (NeurIPS 2016) measured this “effective receptive field” in trained networks and report that influence concentrates in a roughly Gaussian blob much smaller than the theoretical box, growing with the square root of depth rather than linearly. So when a design guide says “make sure the receptive field covers the object,” treat the arithmetic above as an upper bound, not a guarantee.

This is also why small kernels won the architecture argument. Early networks used 11x11 and 7x7 kernels in their first layers; the VGG networks (2014) showed that stacking 3x3 kernels achieves the same receptive field with fewer parameters and more interleaved nonlinearities: two 3x3 layers (\(2 \cdot 9 \cdot C^2\) weights, for \(C\) channels) cover the 5x5 receptive field of one 5x5 layer (\(25 \cdot C^2\) weights) with an extra ReLU in between. Nearly every architecture since is built from 3x3 or smaller kernels; the trade-offs are taken up in Section 2.2.

The hierarchy this produces is not just a computational trick — it shapes what networks learn. First-layer kernels in trained networks consistently come out as oriented edge and color-blob detectors; middle layers respond to textures and simple parts; deeper layers to object-like configurations. That layered structure is what makes the features reusable across tasks, the fact that transfer learning exploits.

Channels: the full convolutional layer

Real inputs have a channel dimension (3 for RGB, or however many feature maps the previous layer produced), and PyTorch tensors carry it as (N, C, H, W) with N the batch size. Channels-first is a framework convention rather than a law: TensorFlow and most image libraries default to channels-last (H, W, C), while PyTorch standardized on the layout its Torch and cuDNN ancestry optimized first, so an image arriving from a loader usually needs x.permute(2, 0, 1): an axis reordering, never a reshape, which would keep the flat element order and silently scramble which value belongs to which pixel. The full operation:

  • One kernel has shape (C_in, kh, kw): it takes an inner product across all input channels at each position, producing one output map. A “vertical edge in red minus green” detector is expressible only because the kernel spans channels.
  • A conv layer holds C_out such kernels plus C_out biases (weight shape (C_out, C_in, kh, kw)) and stacks their outputs into a (N, C_out, H_out, W_out) tensor.

So nn.Conv2d(3, 16, kernel_size=3, padding=1) from the example holds 16 kernels of shape (3, 3, 3): 16 * 27 + 16 = 448 parameters, as printed. Channel width, kernel size, stride, and padding are the entire vocabulary. A special case worth knowing is the 1x1 convolution, a per-position linear layer across channels used to widen, narrow, or mix feature maps. Its exact parameter count is \(C_{\text{in}}C_{\text{out}}+C_{\text{out}}\) when bias is enabled; whether that cost is small depends on the channel widths. With this vocabulary fixed, an architecture is just a sequence of these choices — which is what the next section’s fifty-year tour from LeNet to ResNet is actually about.

People. Yann LeCun pursued convolutional networks from his 1989 Bell Labs work on ZIP code digits through LeNet-5 (1998), through years when the approach was widely considered a niche technique, to its post-2012 dominance. He shared the 2018 Turing Award with Yoshua Bengio and Geoffrey Hinton for deep learning; the standard MNIST benchmark you will meet repeatedly descends from his handwritten-digit work.

Under the hood: convolution is a matrix multiply

The nested Python loops in conv2d_np are honest but slow, and the slowness is not just Python overhead. Even written in C, a direct sliding-window loop makes scattered memory accesses and cannot easily exploit the hardware that modern chips actually provide: extremely fast dense matrix multiplication (GEMM), polished by decades of BLAS-library and, later, GPU engineering. So production frameworks usually do not compute convolution as a sliding window at all. They reshape it into a matrix multiply.

The standard trick is called im2col (“image to columns”). Walk the kernel window over the input once, and copy each patch it covers, all channels included, into one column of a big matrix. A 3-channel input with 3x3 kernels yields columns of height 3 * 3 * 3 = 27, one column per output position. Flatten each kernel of the layer into one row of a second matrix. Then the entire convolutional layer (every kernel, every position) is a single matrix product, and the result only needs reshaping back into feature maps:

import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
torch.manual_seed(0)

def im2col(x, kh, kw):
    """x: (C, H, W) -> matrix of patches (C*kh*kw, out_h*out_w). Stride 1, no padding."""
    C, H, W = x.shape
    out_h, out_w = H - kh + 1, W - kw + 1
    cols = np.zeros((C * kh * kw, out_h * out_w), dtype=x.dtype)
    col = 0
    for i in range(out_h):
        for j in range(out_w):
            cols[:, col] = x[:, i:i + kh, j:j + kw].ravel()
            col += 1
    return cols

x = np.random.randn(3, 8, 8).astype(np.float32)       # 3-channel 8x8 image
w = np.random.randn(16, 3, 3, 3).astype(np.float32)   # 16 kernels, 3x3, 3 channels

cols = im2col(x, 3, 3)          # every 3x3x3 patch, flattened into a column
W_mat = w.reshape(16, -1)       # every kernel, flattened into a row
out = (W_mat @ cols).reshape(16, 6, 6)   # ONE matrix multiply = the whole conv

ref = F.conv2d(torch.from_numpy(x)[None], torch.from_numpy(w))[0].numpy()
print(f"patch matrix {cols.shape}, kernel matrix {W_mat.shape}")
print(f"conv-as-matmul output: {out.shape}, "
      f"max |matmul - F.conv2d| = {np.abs(out - ref).max():.2e}")
print(f"patch matrix holds {cols.size:,} floats vs {x.size:,} in the image "
      f"({cols.size / x.size:.1f}x duplication)")

Running this prints:

patch matrix (27, 36), kernel matrix (16, 27)
conv-as-matmul output: (16, 6, 6), max |matmul - F.conv2d| = 0.00e+00
patch matrix holds 972 floats vs 192 in the image (5.1x duplication)

The displayed difference is 0.00e+00: both implementations compute the same algebraic convolution on this input. That need not imply bitwise identity for every backend, because a GEMM kernel may accumulate products in a different floating-point order. The cost shows in the last line: because a stride-1 window covers each pixel up to kh * kw times, the patch matrix stores those pixels repeatedly — 5.1x duplication here, approaching 9x for 3x3 kernels on large inputs. im2col deliberately spends memory to buy a GEMM call. Libraries mitigate the copy with variants (lowering only tiles at a time, or “implicit GEMM” kernels that index the input in place).

There is a conceptual payoff too, beyond speed. Written as W_mat @ cols, convolution is manifestly a linear operation: a matrix multiply whose matrix is highly structured — sparse (each output touches only a small patch) and weight-tied (the same 27 numbers repeat in every column of W_mat’s interaction with the input). That is exactly the “dense layer with constraints” framing from the start of this section, now literal. It also explains a fact that returns in Section 3.3: convolution’s compute cost per layer is linear in the number of pixels, while self-attention’s grows quadratically with the number of positions attended over.

Historical note. The lowering trick is old enough to predate the deep learning boom. Chellapilla, Puri, and Simard described unrolling convolutions into matrix products for document-recognition CNNs in 2006, reporting a 2.4–3.0x speedup from unrolling plus CPU BLAS routines, and additional gains from early GPUs. When AlexNet’s authors needed a week-long two-GPU training run in 2012, fast convolution-as-GEMM on graphics hardware was the enabling substrate — the same insight, scaled up by the video-game industry’s decade of investment in parallel arithmetic.

Variations: dilated, grouped, and depthwise convolution

The plain convolutional layer has three widely used variants. Each answers a specific engineering question, and together with 1x1 convolutions they form the parts list for every architecture in Section 2.2.

Grouped convolution answers: must every output channel see every input channel? Split the channels into \(g\) groups and run an independent convolution inside each group, and the parameter count divides by \(g\). The idea entered the field sideways: AlexNet split its channels into two groups so the model would fit across two GPUs, a hardware workaround. Years later, ResNeXt (Xie et al., CVPR 2017) showed that many narrow groups (the paper calls the group count “cardinality”) is a genuinely useful design axis, improving accuracy at matched parameter budgets.

Depthwise convolution is the extreme case \(g = C\): every channel is filtered by its own single-channel kernel, and no channel mixing happens at all. On its own that is too restrictive (features could never combine), so it is nearly always paired with a 1x1 convolution that does the cross-channel mixing. The pair (depthwise 3x3, then pointwise 1x1) is called a depthwise-separable convolution, and it factorizes the standard layer’s job (“filter spatially and mix channels, jointly”) into two cheap sequential steps. MobileNets (Howard et al., 2017) built an entire architecture family on this factorization for phone-scale compute, and Xception (Chollet, CVPR 2017) used it at server scale.

Dilated (atrous) convolution answers: can the receptive field grow without pooling away resolution? Insert gaps between the kernel’s taps (dilation d places taps d pixels apart), so a 3x3 kernel covers a (3-1)*d + 1 region with the same 9 parameters. Stacking increasing dilations grows the receptive field exponentially while keeping full spatial resolution, which is exactly what dense-prediction tasks like semantic segmentation need; Yu and Koltun made the case at ICLR 2016. The same idea, in one dimension, powers WaveNet’s audio models (van den Oord et al., 2016), where dilations doubling layer by layer let a convolutional stack span thousands of time steps.

All three, measured:

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

def n_params(m):
    return sum(p.numel() for p in m.parameters())

# Four ways to map 64 channels to 64 channels with 3x3 spatial extent.
standard  = nn.Conv2d(64, 64, 3, padding=1, bias=False)
grouped   = nn.Conv2d(64, 64, 3, padding=1, groups=4, bias=False)
depthwise = nn.Conv2d(64, 64, 3, padding=1, groups=64, bias=False)
pointwise = nn.Conv2d(64, 64, 1, bias=False)

print(f"standard 3x3, 64->64:           {n_params(standard):6,d} params")
print(f"grouped 3x3 (4 groups):         {n_params(grouped):6,d} params")
print(f"depthwise 3x3 (64 groups):      {n_params(depthwise):6,d} params")
print(f"depthwise separable (dw + 1x1): {n_params(depthwise) + n_params(pointwise):6,d} params")

x = torch.randn(2, 64, 16, 16)
y = pointwise(depthwise(x))
print(f"depthwise-separable output: {tuple(y.shape)} (same as standard: "
      f"{tuple(standard(x).shape)})")

# Dilation: spread the 9 kernel taps apart without adding parameters.
img = torch.zeros(1, 1, 9, 9)
img[0, 0, 4, 4] = 1.0                    # one bright pixel in the center
k = torch.ones(1, 1, 3, 3)               # 9 taps, always 9 parameters
for d in [1, 2, 3]:
    out = F.conv2d(img, k, padding=d, dilation=d)
    hit = (out[0, 0] > 0).nonzero()
    span = hit[:, 0].max().item() - hit[:, 0].min().item() + 1
    print(f"dilation {d}: responses span {span}x{span} "
          f"-> effective kernel size {(3 - 1) * d + 1}")

Running this prints:

standard 3x3, 64->64:           36,864 params
grouped 3x3 (4 groups):          9,216 params
depthwise 3x3 (64 groups):         576 params
depthwise separable (dw + 1x1):  4,672 params
depthwise-separable output: (2, 64, 16, 16) (same as standard: (2, 64, 16, 16))
dilation 1: responses span 3x3 -> effective kernel size 3
dilation 2: responses span 5x5 -> effective kernel size 5
dilation 3: responses span 7x7 -> effective kernel size 7

The parameter arithmetic is worth internalizing. Four groups cut the standard layer’s 36,864 weights to exactly a quarter (9,216), because each output channel now spans 16 input channels instead of 64. Full depthwise is 64x cheaper (576). The depthwise-separable pair lands at 4,672, about 7.9x cheaper than standard. Its cost ratio to a standard \(k \times k\) convolution comes from one division: per output position the pair spends \(C_{\text{in}} k^2\) multiplies (depthwise) plus \(C_{\text{in}} C_{\text{out}}\) (pointwise) against the standard layer’s \(C_{\text{in}} C_{\text{out}} k^2\), and the quotient simplifies to \(1/C_{\text{out}} + 1/k^2\); the reciprocal is the speed/parameter reduction factor, approaching 9x for wide 3x3 layers. Dilation changes the connectivity and receptive field with zero added parameters: the same 9 taps are spread over 3x3, 5x5, then 7x7. Grouped, depthwise, and dilated convolutions impose different structural constraints; they are not ordered by one universal notion of expressive power.

Pitfall. Two traps hide in these variants. First, groups must divide both channel counts; nn.Conv2d(64, 96, 3, groups=4) works (64/4 in, 96/4 out per group) but mismatched counts raise an error only at layer construction, which can be far from the config file that caused it. Second, a depthwise layer without a following 1x1 mixes nothing across channels — a network of only depthwise convs processes 64 channels as 64 independent grayscale pipelines forever. If a “separable” network underperforms wildly, check that the pointwise halves did not get dropped in refactoring.

Going deeper

The clean story of this section (local windows, shared weights, growing receptive fields) has refinements that matter in research settings, and a few places where it oversimplifies. The five notes below are research frontiers, not quiz material.

Equivariance can be engineered for more than translation. Parameter sharing across positions is what makes convolution translation-equivariant; nothing in the mathematics restricts the idea to translations. Group-equivariant CNNs (Cohen and Welling, ICML 2016) share weights across rotations and reflections too, so that a rotated input produces a correspondingly transformed feature map. For data with strong rotational symmetry (microscopy, aerial imagery, molecular structures), such architectures encode the symmetry directly instead of hoping augmentation (Section 2.3) teaches it.

The invariance story has a hole: aliasing. Strided convolution and pooling subsample feature maps, and subsampling without a low-pass filter aliases — meaning a one-pixel input shift can change the subsampled feature map in ways no later layer can undo. Zhang (ICML 2019) showed that classic CNNs are measurably not shift-invariant for this reason, and that inserting a small blur before each subsampling step (anti-aliasing, straight from signal processing) improves both measured shift-consistency and ImageNet accuracy. The paper’s title, “Making Convolutional Networks Shift-Invariant Again,” is a fair summary of how the field received it.

Convolutions know more about position than the story admits. Zero-padding marks the image border: a unit whose window hangs over the edge sees a distinctive wall of zeros, and networks learn to use it. Islam, Jia, and Bruce (ICLR 2020) demonstrated that CNNs can read out absolute position surprisingly well from padding effects alone. Perfect translation equivariance, in other words, holds in the interior and quietly fails near borders — occasionally the explanation for weird edge-of-image behavior in deployed models.

The speed story keeps evolving. Beyond im2col-GEMM, libraries lower convolutions through FFTs (efficient for large kernels) and Winograd’s minimal-filtering algorithms (Lavin and Gray, CVPR 2016), which cut the multiplications needed for the small 3x3 kernels that dominate modern networks. Backends autotune among these per layer shape — one reason two runs of the same model can differ at the last decimal place on GPU, a reproducibility wrinkle that resurfaces in Section 2.5.

And the priors themselves are now a design choice, not a law. Vision Transformers (Section 3.8) weaken convolution’s built-in locality and translation equivariance: self-attention may connect distant patches immediately, while learned projection and Transformer weights are still shared across positions. Positional encodings then identify locations. This less vision-specific prior can require more data or augmentation than a comparable CNN, especially at small scale.

Common pitfalls

  • Shapes off by a little (for example 6x6 where you expected 8x8). Cause: forgetting that unpadded convolution shrinks the map by kernel - 1 per layer. Fix: apply the output-size formula \(\lfloor(\text{in} + 2p - k) / s\rfloor + 1\) layer by layer, and use padding = k // 2 (odd k) to preserve size.
  • Your convolution “matches PyTorch” only for symmetric kernels. Cause: your implementation flips the kernel (true convolution) while conv2d does not; symmetric test kernels hide the difference. Fix: test with an asymmetric kernel, as our verification script effectively does with random kernels.
  • RuntimeError about expected 4 dimensions. Cause: nn.Conv2d requires (N, C, H, W), and a single grayscale image loaded as (H, W) is missing two dimensions. Fix: add them (x.view(1, 1, H, W) or x[None, None]), and keep channel order straight — torchvision loads images channels-first, matplotlib wants channels-last.
  • Model behaves differently when an object moves a few pixels. Cause: expecting exact translation invariance; conv layers are equivariant, pooling gives only partial tolerance, and the final dense layers can break even that. Fix: rely on augmentation with shifts (Section 2.3) and global average pooling rather than assuming invariance you never built in.
  • A receptive-field calculation says the network “sees” the object, yet it ignores context just outside a small neighborhood. Cause: the theoretical receptive field is an upper bound; effective influence concentrates near the center and grows more slowly with depth. Fix: when context matters, add downsampling stages or dilation rather than counting on distant pixels inside a nominally covered region, and verify with the gradient-support measurement from this section.

Try it yourself

  1. Extend conv2d_np to multi-channel input and multiple kernels: input (C_in, H, W), weights (C_out, C_in, kh, kw), output (C_out, H_out, W_out). Verify against F.conv2d with random tensors, as in the first example. This is most of HW2’s first problem.
  2. Build the horizontal-edge analog of edge_k and apply both kernels to an image containing one vertical and one horizontal edge. Confirm each detector fires only on its own orientation.
  3. Compute the receptive field of this stack by hand: 3x3 conv, 2x2 max pool (stride 2), 3x3 conv, 2x2 max pool (stride 2), 3x3 conv. Then verify empirically: feed two images through that differ in exactly one pixel and see which output positions change.
  4. Time conv2d_np against F.conv2d on a 128x128 input. The gap you observe is why real frameworks lower convolution to matrix multiplication (im2col) or specialized kernels instead of Python loops.

Further reading

  • Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, 1998. https://ieeexplore.ieee.org/document/726791 — the LeNet-5 paper; Section II is still one of the clearest first-principles cases for convolution and subsampling.
  • K. Fukushima, “Neocognitron: A Self-organizing Neural Network Model…,” Biological Cybernetics, 1980. https://link.springer.com/article/10.1007/BF00344251 — the pre-backprop ancestor: alternating feature-detector and tolerance layers, directly inspired by Hubel and Wiesel.
  • Dive into Deep Learning, Chapter 7: Convolutional Neural Networks. https://d2l.ai/chapter_convolutional-neural-networks/ — a second runnable treatment of every operation in this section, with executable notebooks.
  • K. Simonyan, A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition” (VGG), ICLR 2015. https://arxiv.org/abs/1409.1556 — the paper that settled the small-kernels argument; see its Section 2.3 for the stacked-3x3 receptive-field analysis.
  • F. Yu, V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” ICLR 2016. https://arxiv.org/abs/1511.07122 — dilation as a way to grow context exponentially without losing resolution, motivated by segmentation.
  • A. Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv 2017. https://arxiv.org/abs/1704.04861 — the depthwise-separable factorization worked into a full architecture family, with the cost accounting spelled out.
  • K. Chellapilla, S. Puri, P. Simard, “High Performance Convolutional Neural Networks for Document Processing,” IWFHR 2006. https://inria.hal.science/inria-00112631 — the unrolled-convolution (im2col) trick and early GPU speedups, years before the deep learning boom.
  • W. Luo, Y. Li, R. Urtasun, R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” NeurIPS 2016. https://arxiv.org/abs/1701.04128 — why real influence is a Gaussian blob far smaller than the receptive-field arithmetic suggests.
  • R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML 2019. https://arxiv.org/abs/1904.11486 — the aliasing hole in the invariance story, and the classical anti-aliasing fix.
  • PyTorch nn.Conv2d documentation. https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html — the authoritative statement of shapes, padding modes, groups, dilation, and the cross-correlation convention.

Before class you should be able to

  • Compute the output size of a conv layer from input size, kernel, stride, and padding — and pick the padding that preserves size.
  • Count the parameters of a conv layer from its weight shape plus biases (e.g. why nn.Conv2d(3, 16, 3) costs 448), and explain why the count is independent of image size.
  • Explain the two structural bets (locality and parameter sharing) and what each buys over a dense layer.
  • Compute the receptive field of a stack of 3x3 convs, with and without stride-2 downsampling in between.
  • Say what max pooling buys (tolerance to position, not true invariance) and what a 1x1 convolution does.
  • State the cross-correlation convention and when it matters.
  • Read conv2d_np-style code and predict the shapes it produces.

Connects to

  • Quiz 1 takes the final 30 minutes of this class (6:45–7:15 PM, in Canvas), covering Module 1 (S01–S08). This session is itself covered by Quiz 2, given at S13.
  • HW2 (out at S12, due at S18) has you implement convolution by hand and verify it against PyTorch — exactly this section’s first script, extended to channels.
  • Sections 2.2 and 2.3, the next two readings, are built entirely from this section’s vocabulary.

The PDF posted in Canvas is the assigned version of this reading; it is built from this page.

← S08 · PyTorch in Practice · All readings · S10 · From LeNet to ResNet →