Session S07

Regularization, Initialization, Normalization

Control fitting · choose signal scale · identify normalization groups

Deep Learning · CS 4715/5715 · University of Idaho

Match the tool to the observed problem

Parameter count alone does not determine generalization. These tools address different aspects of training.

Weight decay from S06; now measure its effect

\[W^{\mathrm{new}}=(1-\eta\lambda)W\;-\;\eta\cdot\text{adaptive step}\]

• The penalty added to the loss is \(R(W)=\frac\lambda2\lVert W\rVert_F^2=\frac\lambda2\sum_{ij}W_{ij}^2\), the sum of squared entries; \(\lambda\) sets its strength.
• Its matrix gradient is \(\nabla_W R=\lambda W\), differentiated one entry at a time on the next page.
• Smaller weights do not guarantee better predictions on new data.

The mechanism is established. Next: vary the decay strength and measure validation error.

Differentiate the penalty one weight at a time

Deriving: why \(\nabla_W R=\lambda W\). S06 stated this result; here it is derived entry by entry as reinforcement, for \(W\in\mathbb R^{2\times2}\).

\[R(W)=\frac\lambda2\left(W_{11}^2+W_{12}^2+W_{21}^2+W_{22}^2\right)\]

Choose \(W_{12}\). Hold the other three weights fixed; their terms contribute zero.

\[\frac{\partial R}{\partial W_{12}}=\frac\lambda2(2W_{12})=\lambda W_{12}\]

\[\nabla_W R=\begin{bmatrix}\lambda W_{11}&\lambda W_{12}\\\lambda W_{21}&\lambda W_{22}\end{bmatrix}=\lambda W\]

Result: each entry’s partial derivative is \(\lambda\) times that entry, so the matrix of partials is \(\lambda W\). The matrix rule is shorthand for one scalar derivative per weight.

A small dataset can be fitted more closely than it should

30 noisy points from \(y=\sin(2x)\) · two 128-unit tanh layers · AdamW, lr = 0.001 · 4000 updates.

Validation uses the known noise-free function in this synthetic experiment.

Both final metrics are measured after the last update. Training targets contain noise; validation targets here do not.

Compare the same validation metric across checkpoints

No-decay run: the best recorded checkpoint is much better than the final one.

The no-decay run overfits: validation error ends far above its best value, 0.2335 against 0.0275. Choose a checkpoint by validation. This does not show that every later update only memorizes noise.

Choose decay and stopping time together

Recording a best score does not save its weights. Reserve separate test data for the final evaluation.

Dropout multiplies the activation matrix by a random mask

\[X,M,Y\in\mathbb R^{N\times C},\qquad M_{ij}\sim\operatorname{Bernoulli}(1-p)\]

\(X\) holds the activations of \(N\) samples with \(C\) features each, \(M\) is a fresh binary mask of the same shape, and \(p\) is the drop probability.

\[Y_{\mathrm{train}}=(X\odot M)/(1-p),\qquad Y_{\mathrm{eval}}=X\]

One draw with \(N=C=2\) and \(p=0.5\), mask first, then rescale:

\[X=\begin{bmatrix}1&1\\1&1\end{bmatrix},\ M=\begin{bmatrix}1&0\\0&1\end{bmatrix}\ \Rightarrow\ X\odot M=\begin{bmatrix}1&0\\0&1\end{bmatrix}\ \Rightarrow\ Y=\frac{X\odot M}{1-p}=\begin{bmatrix}2&0\\0&2\end{bmatrix}\]

One entry is kept with probability \(1-p\) and then multiplied by \(1/(1-p)\), so its expected output is \((1-p)\cdot\frac{1}{1-p}\cdot1=1\), its value before dropout. A fresh mask is drawn at every training step; this one is illustrative.

Dropout code: the mode changes the operation

drop = nn.Dropout(p=0.5)
x = torch.ones(100, 100)
drop.train()
y_train = drop(x)
drop.eval()
y_eval = drop(x)

Dropout: follow one activation through the random mask

Deriving: the expected training-mode output of one activation, for \(x=1\) and \(p=0.5\).

\[\mathbb E[y]=0.5\times0+0.5\times2=1\]

Result: \(\mathbb E[y]=x\). Training preserves this activation’s expectation, not every sampled value. In evaluation mode, dropout returns the input.

Initialization must break symmetry and set a useful scale

• Identical hidden units with identical incoming and outgoing parameters can keep receiving identical updates.
• Random weights are a practical way to break this symmetry; biases can often start at zero.
• After breaking symmetry, choose the weight scale to keep signals usable through depth.

Use a matrix initialization rule, inspect its forward signals, and map the choice to code. The derivation pages explain the scale.

Initialize the weight matrix for its input width and activation

Each weight entry is drawn independently with zero mean. fan-in \(D\) is the number of inputs feeding one output unit.

nn.init.kaiming_normal_(layer.weight, mode="fan_in", nonlinearity="relu")

Here layer is an nn.Linear(D, H) followed by ReLU. The rule samples the whole weight matrix; the next four pages derive why the ReLU variance is \(2/D\).

The per-layer factor compounds through depth

Write \(q_\ell=\mathbb E[x^2]\) for the average squared activation at layer \(\ell\). A stack of identical layers multiplies it by the same factor \(f\) at every layer:

\[q_L=f^{\,L}q_0\]

The initialization question is therefore one question about \(f\): choose the weight variance \(\sigma^2\) so that \(f\approx1\). The next two pages compute \(f\) for a linear layer and for ReLU.

One neuron adds contributions from its inputs

Page 12 uses \(\sigma^2=2/D\) for ReLU layers. To derive this, first compute the factor \(f\) for a linear layer alone: one unit with \(D\) inputs, weight variance \(\sigma^2\), and input second moment \(q=\mathbb E[x_i^2]\).

\[z=w_1x_1+\cdots+w_Dx_D\]

Assume equal input second moments, zero bias, and independent zero-mean weights independent of the inputs. Square and expand:

\[z^2=\sum_i w_i^2x_i^2+\sum_{i\neq j}w_ix_iw_jx_j\]

The first sum has \(D\) diagonal terms; the second has \(D(D-1)\) cross terms. We take expectations next.

Cross terms vanish, diagonal terms give the factor

Cross terms vanish: \(\mathbb E[w_ix_iw_jx_j]=\mathbb E[w_i]\cdot\mathbb E[x_iw_jx_j]=0\) because \(w_i\) is independent and zero-mean.

Diagonal terms factor: \(\mathbb E[w_i^2x_i^2]=\mathbb E[w_i^2]\,\mathbb E[x_i^2]=\sigma^2q\). Sum \(D\) of them:

\[\mathbb E[z^2]=D\sigma^2q\]

Result: a linear layer multiplies the average square by \(f=D\sigma^2\). Without an activation, \(\sigma^2=1/D\) balances the layer. Page 16 adds ReLU's correction to reach \(2/D\).

ReLU halves the average square for a symmetric input

The linear layer alone gives \(f=D\sigma^2\) (page 14). ReLU follows it, so we also need the factor ReLU applies to the average square, for a pre-activation distribution symmetric about zero.

Symmetry pairs each value with its negative. ReLU sends one member of every pair to zero and leaves the other unchanged, so half the squared values survive.

\[\mathbb E[\operatorname{ReLU}(z)^2]=\tfrac12\mathbb E[z^2]\quad\text{for symmetric }z\]

Result: ReLU contributes a factor of \(\tfrac12\). The output mean is positive, so the variance does not halve; it is the second moment that does.

Choose weight variance to balance the per-layer factor

Multiply the two factors: the \(\sigma^2\) giving \(f=1\) for a linear layer plus ReLU is page 12’s variance:

\[q_{\mathrm{next}}\approx\underbrace{D\sigma^2}_{\text{linear}}\times\underbrace{\tfrac12}_{\text{ReLU}}\times\,q=\frac{D\sigma^2}{2}q\]

\[\frac{D\sigma^2}{2}=1\quad\Rightarrow\quad\sigma^2=\frac2D\quad\text{(Kaiming, fan-in)}\]

Result: \(\sigma^2=2/D\) holds \(f\) near 1 for ReLU layers. A factor below or above 1 compounds across depth, which the next experiment measures.

The 40-layer experiment: what is run and what is predicted

Forward activations only: no gradients, no claim about trainability. The next page measures these factors.

Observe the signal through 40 ReLU layers

Measured factors are geometric means over layers 1 to 40 and match the predictions to within 1%. Kaiming’s 0.7% per-layer shortfall compounds to 0.75 across 39 layers: finite-network drift.

Identify the input axis before initializing a weight matrix

layer = nn.Linear(in_features=256, out_features=256)
nn.init.kaiming_normal_(
    layer.weight, mode="fan_in", nonlinearity="relu")
nn.init.zeros_(layer.bias)

fan_in counts inputs to one output. Here fan-in = 256, so each weight is drawn from \(\mathcal N(0,\;2/256)\), i.e. \(\sigma\approx0.088\).

Normalization starts by choosing a group of values

We will use the same two samples and two features for both methods:

\[X=\begin{bmatrix}1&2\\3&6\end{bmatrix}\]

BatchNorm1d in training groups each column. LayerNorm over the two features groups each row.

\(X\in\mathbb R^{N\times C}\): rows are samples; columns are features. Here \(N=C=2\).

BatchNorm1d: reduce over rows, then broadcast over rows

\[\mu_B,s_B^2\in\mathbb R^{1\times C},\qquad\widehat X=(X-\mu_B)\oslash\sqrt{s_B^2+\epsilon}\]

mu = x.mean(dim=0, keepdim=True)
var = x.var(dim=0, unbiased=False, keepdim=True)
x_hat = (x - mu) / torch.sqrt(var + 1e-5)

For this X: \(\mu_B=[2,4]\), \(s_B^2=[1,4]\). With scale 1 and shift 0, \(\widehat X\approx\left[\begin{smallmatrix}-1&-1\\1&1\end{smallmatrix}\right]\).

LayerNorm: reduce over features, then broadcast within each row

\[\mu_L,s_L^2\in\mathbb R^{N\times1},\qquad\widehat X=(X-\mu_L)\oslash\sqrt{s_L^2+\epsilon}\]

mu = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, unbiased=False, keepdim=True)
x_hat = (x - mu) / torch.sqrt(var + 1e-5)

For this X: \(\mu_L=[1.5,4.5]^\top\), \(s_L^2=[0.25,2.25]^\top\); \(\widehat X\approx\left[\begin{smallmatrix}-1&1\\-1&1\end{smallmatrix}\right]\).

Learned scale and shift broadcast across the normalized matrix

Normalization fixes each group to zero mean and unit variance, which also removes any scale or shift the next layer might need. Learned \(\gamma\) and \(\beta\) let the network restore them, so the layer keeps its expressive range.

\[Y=\widehat X\odot\gamma+\beta,\qquad X,Y\in\mathbb R^{N\times C}\]

y = x_hat * gamma + beta  # gamma and beta have shape (C,)

Default layers learn scale and shift. The demonstrations disable this affine transform to isolate the group calculation.

The same sample can change when its batchmates change

Reading example: 8 samples, 4 features; compare sample 0 with itself in a smaller batch.

Sample 0 did not change. Why did its BatchNorm output change while its LayerNorm output did not?

BatchNorm’s column statistics come from the batch, and the batch changed. LayerNorm(C) uses only sample 0’s own four features, which are the same in both batches, so its output is identical to every printed digit.

Train and eval select different behavior in some layers

The mode changes what a layer computes. Dropout draws a fresh random mask and rescales in training, and is the identity in evaluation. BatchNorm with default tracking normalizes by the current batch’s statistics in training and updates its running estimates; in evaluation it normalizes by those stored estimates, so its output no longer depends on batchmates. LayerNorm computes the same thing in both modes, because its statistics come from the item itself.

Read the layer configuration and evaluation calls

bn = nn.BatchNorm1d(4, affine=False)
ln = nn.LayerNorm(4, elementwise_affine=False)
bn.train()
y_bn = bn(x)       # x has shape (8, 4)
y_ln = ln(x)
bn.eval()
with torch.no_grad():
    y_eval = bn(x) # use the running statistics

affine=False isolates the statistics in this demonstration. Default layers normally learn scale and shift.

model.eval() selects evaluation behavior. no_grad() controls gradient recording. Use both for ordinary evaluation.

Read the arrays, choose the axes, identify the code

The derivation pages explain why the supplied gradient and initialization rules hold. Use their self-checks during home review.

Three quick checks

Why does the L2 gradient have W’s shape? Which quantity does symmetric-input ReLU halve? To normalize \(X_{0,0}\) in our 2×2 X, which entries does each method group together?

L2 penalty is \(\tfrac\lambda2\sum_{ij}w_{ij}^2\). Differentiating with respect to each \(w_{ij}\) gives \(\lambda w_{ij}\), one scalar per weight, so the gradient \(\lambda W\) has the same shape as \(W\).

ReLU halves the second moment \(\mathbb E[z^2]\), not generally the variance. After ReLU the mean is positive, so \(\operatorname{Var}=\mathbb E[z^2]-(\mathbb E[z])^2\) does not simply halve.

The top-left entry is row 0, column 0 of \(X=\bigl[\begin{smallmatrix}1&2\\3&6\end{smallmatrix}\bigr]\). BatchNorm groups the column: normalize 1 using \(\{1,3\}\). LayerNorm groups the row: normalize 1 using \(\{1,2\}\).

For next time

• Read: S08: PyTorch in Practice.
• S08: assemble nn.Module, optimizer, data loading, and the training loop.
• HW1 is due at S08: micro-autograd; 12-hour grace period, then the syllabus late policy.
• Quiz 1 is at S09: Module 1. The assigned S07 reading also covers augmentation and wider normalization choices.