Backpropagation II反向传播 II

Input gradients stay with their own examples输入梯度分别留在各自的样本

Same values: \(w=2,\ dZ_1=1,\ dZ_2=2\). Each input has local derivative \(w\).沿用 \(w=2,\ dZ_1=1,\ dZ_2=2\)。每个输入对应的局部导数都是 \(w\)。

\(x_1\) and \(x_2\) are different nodes. Keep \(2\) and \(4\) separate.\(x_1\)、\(x_2\) 是不同节点，梯度 \(2\)、\(4\) 分别保存。

\[dX_{ni}=\sum_j dZ_{nj}W_{ij}=(dZ\,W^\top)_{ni}\]

With several outputs, one input reaches every output column \(j\): sum over \(j\), within the same example.有多个输出时，一个输入影响每个输出列 \(j\)：在同一样本内对 \(j\) 求和。

A shared bias collects contributions from both rows共享偏置收集两行的贡献

\(z_n=x_nw+b\). Even at \(b=0\), the local derivative with respect to \(b\) is \(1\).\(z_n=x_nw+b\)。即使当前 \(b=0\)，输出对 \(b\) 的局部导数仍为 \(1\)。

\[db=1+2=3\]

\[db_j=\sum_n dZ_{nj},\qquad \operatorname{shape}(db)=(H,)\]

One bias per output column, reused across examples: broadcast forward, sum backward.每个输出列有一个偏置，在不同样本间共享：前向广播，反向求和。

Two-layer classifier: the forward graph两层分类网络：前向计算图

Two trainable layers: \((W_1,b_1)\) and \((W_2,b_2)\). ReLU follows the first affine transformation.两组可训练层：\((W_1,b_1)\) 与 \((W_2,b_2)\)。第一层仿射变换之后使用 ReLU。

The diagram shows one example. Repeat the same network for all \(N\) rows; \(L\) averages their losses.图中画出一个样本。对全部 \(N\) 行使用同一个网络；\(L\) 对各行损失取平均。

Forward arrays, formulas, and shapes前向数组、计算公式与形状

\(N\): examples · \(D\): input features · \(H\): hidden units · \(C\): classes.\(N\)：样本数 · \(D\)：输入特征数 · \(H\)：隐藏单元数 · \(C\)：类别数。

Parameter shapes: \(W_1:(D,H),\ b_1:(H,),\ W_2:(H,C),\ b_2:(C,)\).参数形状：\(W_1:(D,H),\ b_1:(H,),\ W_2:(H,C),\ b_2:(C,)\)。

The output layer: softmax and cross-entropy, one row at a time输出层：softmax 与交叉熵，一次一行

\[P_{nc} = \frac{e^{Z_{2,nc}}}{\sum_{c'} e^{Z_{2,nc'}}} \qquad \ell_n = -\log P_{n,\,y_n} \qquad L = \frac{1}{N}\sum_{n} \ell_n\]

\[\ell_n = -Z_{2,ny_n} + \log \sum_{c'} e^{Z_{2,nc'}}\]

• Softmax turns the \(C\) scores of row \(n\) into probabilities that sum to 1; the loss of the row is the negative log of the probability given to the true class \(y_n\); \(L\) averages the \(N\) rowssoftmax 把第 \(n\) 行的 \(C\) 个得分变成和为 1 的概率；这一行的损失是真实类别 \(y_n\) 所得概率的负对数；\(L\) 对 \(N\) 行取平均
• The second line is the same loss written with the scores: the true-class score, and a log-sum-exp. Its derivative is the next page第二行是同一个损失用得分写出来：真实类别的得分，加一个 log-sum-exp。它的导数在下一页

Backward starts at the loss: the gradient of the logits从损失开始反向：对类别得分求导

Backward route: \(L\ \longrightarrow\ dZ_2\ \longrightarrow\ dA_1\ \longrightarrow\ dZ_1\).反向路线：\(L\ \longrightarrow\ dZ_2\ \longrightarrow\ dA_1\ \longrightarrow\ dZ_1\)。

\[\frac{\partial\ell_n}{\partial Z_{2,nc}}=\underbrace{-\mathbf1[c=y_n]}_{\text{true-class score}}+\underbrace{P_{nc}}_{\text{log-sum-exp}}\]

\[dZ_2=\frac{\partial L}{\partial Z_2}=\frac{P-Y}{N},\qquad Y=\operatorname{onehot}(y)\]

Shape: \(P,\ Y,\ dZ_2\in\mathbb R^{N\times C}\): \(N\) rows, \(C\) columns.形状：\(P,\ Y,\ dZ_2\in\mathbb R^{N\times C}\)：\(N\) 行、\(C\) 列。

Subtract 1 only at each true class. Divide by N once because the loss averages examples.只在每行的真实类别处减 1。损失对样本取平均，因此在这里除一次 N。

Layer 2 backward: three destinations for the gradient第 2 层反向：梯度传向三个位置

Local forward operation: \(Z_2=A_1W_2+b_2\). Incoming gradient: \(dZ_2\).当前的前向运算：\(Z_2=A_1W_2+b_2\)。传入梯度：\(dZ_2\)。

• Weight and bias gradients sum contributions across examples. \(A_1\) is this layer’s input.权重和偏置梯度对样本贡献求和；\(A_1\) 是这一层的输入。
• \(dA_1\) continues upstream into ReLU. It is a gradient, not a parameter update.\(dA_1\) 继续向前传给 ReLU。它是梯度，不是参数更新。

ReLU backward: match each gradient with its maskReLU 反向：每个梯度乘对应的掩码

Forward: \(A_1=\max(0,Z_1)\). Backward: \(dZ_1=dA_1\odot\mathbf1[Z_1>0]\).前向：\(A_1=\max(0,Z_1)\)。反向：\(dZ_1=dA_1\odot\mathbf1[Z_1>0]\)。

At zero, the ordinary derivative does not exist; this backward rule chooses \(0\).零点不存在普通导数；这里的反向规则约定取 \(0\)。

All three arrays have shape \((N,H)\). \(\odot\) means elementwise multiplication.三个数组的形状都是 \((N,H)\)。\(\odot\) 表示逐元素相乘。

Layer 1 backward and the saved forward values第 1 层反向与保存的前向值

Local forward operation: \(Z_1=XW_1+b_1\). Incoming gradient: \(dZ_1\).当前的前向运算：\(Z_1=XW_1+b_1\)。传入梯度：\(dZ_1\)。

Saved values: \(X,A_1\) for weight gradients; \(Z_1\) for the mask; \(P\) for \(dZ_2\); \(W_2\) for \(dA_1\).保存值：权重梯度用 \(X,A_1\)；掩码用 \(Z_1\)；\(dZ_2\) 用 \(P\)；\(dA_1\) 用 \(W_2\)。

The implementation below returns the four parameter gradients; its input data are fixed.接下来的实现返回四个参数梯度，输入数据视为固定。

NumPy 1: build the forward arraysNumPy 1：计算前向数组

Inside forward_backward(params, X, y); params stores W1, b1, W2, b2.位于 forward_backward(params, X, y) 内；params 保存 W1、b1、W2、b2。

W1, b1 = params["W1"], params["b1"]
W2, b2 = params["W2"], params["b2"]
N = X.shape[0]

Z1 = X @ W1 + b1

A1 = np.maximum(0, Z1)

Z2 = A1 @ W2 + b2

@ multiplies matrices; each bias broadcasts across the N rows.@ 执行矩阵乘法；每个偏置向量广播到 N 行。

NumPy 2: stable log-probabilities and mean lossNumPy 2：稳定的对数概率与平均损失

\[s_{nc}=Z_{2,nc}-\max_{c'}Z_{2,nc'},\qquad \log P_{nc}=s_{nc}-\log\sum_{c'}e^{s_{nc'}}\]

Z2s = Z2 - Z2.max(axis=1, keepdims=True)
logp = Z2s - np.log(np.exp(Z2s).sum(axis=1, keepdims=True))
loss = -logp[np.arange(N), y].mean()

• axis=1: operate across classes in each example. keepdims=True: retain shape (N, 1) for row-wise broadcasting.axis=1：在每个样本内沿类别计算。keepdims=True：保留 (N, 1) 形状，便于逐行广播。
• Subtracting a row maximum preserves softmax probabilities and avoids positive exponential overflow.减去每行最大值不改变 softmax 概率，并避免指数的正向溢出。
• Paired indices select each true-class log-probability; the minus sign and mean implement the loss.成对索引选出每行真实类别的对数概率；负号与 mean 实现损失。

NumPy 3: form the starting gradientNumPy 3：构造反向的起点梯度

\[dZ_2=(P-Y)/N\]

dZ2 = np.exp(logp)
dZ2[np.arange(N), y] -= 1
dZ2 /= N

y = [2, 0] → row indices [0, 1] → selected entries (0, 2) and (1, 0).y = [2, 0] → 行索引 [0, 1] → 选中 (0, 2) 和 (1, 0)。

The index example uses 2 rows and 3 classes. All later gradients already carry the 1/N factor.索引示例使用 2 行、3 个类别。后续梯度已经包含 1/N 因子。

NumPy 4: the second affine layerNumPy 4：第二层仿射运算的反向

grads = {}

grads["W2"] = A1.T @ dZ2

grads["b2"] = dZ2.sum(axis=0)

dA1 = dZ2 @ W2.T

• grads stores parameter derivatives under the same keys as params.grads 使用与 params 相同的键，保存参数导数。
• .T transposes a 2-D array. axis=0 sums examples and keeps the class columns..T 转置二维数组。axis=0 对样本求和，保留类别列。
• \(dW_2:(H,C),\ db_2:(C,),\ dA_1:(N,H)\).\(dW_2:(H,C),\ db_2:(C,),\ dA_1:(N,H)\)。

NumPy 5: ReLU, layer 1, and the return valuesNumPy 5：ReLU、第一层与返回值

dZ1 = dA1 * (Z1 > 0)

grads["W1"] = X.T @ dZ1

grads["b1"] = dZ1.sum(axis=0)

return loss, grads

• Z1 > 0 creates a Boolean mask. Multiplication treats False/True as 0/1.Z1 > 0 创建布尔掩码，相乘时 False/True 对应 0/1。
• * is elementwise; @ is matrix multiplication. The function returns a scalar loss and four parameter-gradient arrays.* 逐元素相乘，@ 做矩阵乘法。函数返回标量损失和四个参数梯度数组。

Gradient check: perturb one parameter entry梯度检查：只扰动一个参数元素

Reading example: \(N=8,\ D=3,\ H=4,\ C=2\), float64, random seed 0. Keep the batch and every other parameter fixed.阅读中的例子：\(N=8,\ D=3,\ H=4,\ C=2\)，float64，随机种子 0。固定 batch 和其余全部参数。

name, idx = "W1", (0, 0)
p = params[name]             # the actual array, not a copy
eps = 1e-5
loss, grads = forward_backward(params, X, y)

\[g_{\rm num}=\frac{L(p+\epsilon)-L(p-\epsilon)}{2\epsilon}\]

analytic (backward)   = -0.0713913172
numeric  (two losses) = -0.0713913172

Gradient check code: probe, restore, compare梯度检查代码：扰动、恢复、比较

def numeric_entry(p, idx):
    old = p[idx]
    try:
        p[idx] = old + eps
        lp, _ = forward_backward(params, X, y)
        p[idx] = old - eps
        lm, _ = forward_backward(params, X, y)
    finally:
        p[idx] = old
    return (lp - lm) / (2 * eps)

• p refers to an array in params; changing p[idx] changes what forward reads.p 引用 params 中的数组；修改 p[idx] 就改变了前向读到的参数。
• The underscore discards the recomputed analytic gradients. Only the losses lp and lm form the numerical derivative.下划线丢弃重新算出的解析梯度；数值导数只使用损失 lp、lm。
• finally restores the original value even if evaluation raises an error. No parameter update occurs.finally 即使在计算报错时也恢复原值。这里不执行参数更新。

Gradient check: visit every entry and inspect the errors梯度检查：遍历每个元素并查看误差

for name, p in params.items():
    num = np.zeros_like(p)
    for idx in np.ndindex(p.shape):
        num[idx] = numeric_entry(p, idx)
    ana = grads[name]
    assert ana.shape == p.shape
    rel = np.abs(num - ana) / (np.abs(num) + np.abs(ana) + 1e-12)
    print(name, f"{rel.max():.2e}")

W1 4.32e-10    b1 5.78e-10
W2 7.99e-11    b2 6.01e-12

These maxima summarize all entries of each array. Also inspect absolute error when both gradients are near zero.这些最大值汇总了每个数组的全部元素。两个梯度都接近零时，还要检查绝对误差。

A small discrepancy supports this backward implementation for this forward function and these inputs.误差很小，支持这份反向实现在当前前向函数与输入上的正确性。

Engine design: values, local rules, and execution order引擎设计：数值、局部规则与执行顺序

a, b, c = Value(2.0), Value(3.0), Value(1.0)
p = a * b
q = p + c
y = q.tanh()
y.backward()

Forward builds the graph and saves rules. Backward later calls those rules in dependency order.前向创建计算图并保存规则；反向随后按依赖关系调用规则。

Multiplication 1: create the result and its parent links乘法 1：创建结果与父节点连接

def __mul__(self, other):
    if not isinstance(other, Value):
        other = Value(other)
    out = Value(self.data * other.data,
                (self, other))
    # Attach the local backward rule next.

The forward result is ready, but its gradient is still zero. Its local backward rule has not run.前向结果已经就绪，但它的梯度仍为零，局部反向规则还没有运行。

Multiplication 2: save the two chain-rule contributions乘法 2：保存两份链式法则贡献

Continuation of __mul__; self, other, and out are already defined.继续 __mul__；self、other、out 已经定义。

    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

\[p=ab:\qquad \frac{\partial L}{\partial a}\mathrel{+}=b\,\frac{\partial L}{\partial p},\qquad \frac{\partial L}{\partial b}\mathrel{+}=a\,\frac{\partial L}{\partial p}\]

• def creates a function; it does not run the indented updates.def 创建函数，不执行缩进中的梯度更新。
• Assignment without parentheses saves the function. return out returns the result node.不带括号的赋值保存函数；return out 返回结果节点。
• Each += adds one operand-slot contribution, even when both operands are the same object.每条 += 都累加一个操作数槽的贡献，即使两个操作数是同一个对象。

A closure keeps the operands available for later闭包让反向规则稍后仍能访问操作数

a, b = Value(2.0), Value(3.0)
p = a * b
print(p.data, a.grad, b.grad)  # 6.0  0.0  0.0

p.grad = 1.0
p._backward()                 # call this one local rule
print(a.grad, b.grad)         # 3.0  2.0

These are live object references. Do not mutate operand data between forward and backward.它们是对对象的引用。在前向与反向之间，不要修改操作数的 data。

Addition: two parents, local derivative one加法：两个父节点，局部导数都为 1

def __add__(self, other):
    if not isinstance(other, Value):
        other = Value(other)
    out = Value(self.data + other.data, (self, other))
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

\[q=a+b:\qquad \frac{\partial q}{\partial a}=1,\quad \frac{\partial q}{\partial b}=1\]

The constructor, parent links, closure, assignment and return follow the multiplication pattern.构造输出、连接父节点、定义闭包、保存函数与返回结果，都沿用乘法的结构。

Tanh: one parent and a saved outputtanh：一个父节点与保存的输出

def tanh(self):
    t = math.tanh(self.data)
    out = Value(t, (self,))
    def _backward():
        self.grad += (1.0 - t * t) * out.grad
    out._backward = _backward
    return out

# Also inside Value:
__radd__ = __add__
__rmul__ = __mul__

\[t=\tanh(x),\qquad \frac{dt}{dx}=1-t^2\]

• (self,) is a one-element tuple. The closure reuses the forward output t.(self,) 是只有一个元素的元组，闭包复用前向输出 t。
• Reflected methods support 1 + x and 2 * x; this shortcut uses add/multiply commutativity.反向运算符方法支持 1 + x 和 2 * x；这种简写依赖加法与乘法的交换律。

A gradient ledger for the reused node给复用节点记一张梯度表

a = Value(3.0)
p = a * a
f = p + a

_prev = {a} visits the object once. The two multiply statements still contribute twice._prev = {a} 让遍历只访问该对象一次，但乘法中的两条更新仍然贡献两次。

Backward 1: visit parents before appending a node反向 1：先访问父节点，再把当前节点入列

def backward(self):
    topo, visited = [], set()
    def build(v):
        if v not in visited:
            visited.add(v)
            for parent in v._prev:
                build(parent)
            topo.append(v)
    build(self)

build(f) on f = a*a + a produces topo = [a, p, f].对 f = a*a + a 调用 build(f)，得到 topo = [a, p, f]。

Backward 2: initialize the output and execute in reverse反向 2：初始化输出梯度，倒序执行

Continuation of the same backward(self) method after build(self).继续同一个 backward(self) 方法，接在 build(self) 之后。

    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

\[\frac{\partial f}{\partial f}=1\]

All consumers must contribute to a node before its rule propagates that total into its parents.一个节点的所有使用方都贡献完梯度，它的规则才能把总和继续传给父节点。

Using the engine: a fresh graph for every update使用引擎：每次更新都构建新图

One example: \(x=2,\ y=3\). Model \(\hat y=wx\); loss \(L=(\hat y-y)^2\).一个样本：\(x=2,\ y=3\)。模型 \(\hat y=wx\)，损失 \(L=(\hat y-y)^2\)。

w = Value(1.0)
for step in range(3):
    w.grad = 0.0
    pred = w * 2.0
    error = pred + (-3.0)
    loss = error * error
    loss.backward()
    print(f"{step}: w={w.data:.4f}, L={loss.data:.4f}, dw={w.grad:.4f}")
    w.data -= 0.1 * w.grad

0: w=1.0000, L=1.0000, dw=-4.0000
1: w=1.4000, L=0.0400, dw=-0.8000
2: w=1.4800, L=0.0016, dw=-0.1600

Each printed line is before the update. Rebuild intermediates; keep the parameter object.每一行都在更新之前打印。重新创建中间节点，保留参数对象。

Test local rules, reuse, and graph lifecycle separately分别测试局部规则、复用与图的生命周期

a, b = Value(2.0), Value(3.0)
z = a * b
z.backward()
assert (a.grad, b.grad) == (3.0, 2.0)

a = Value(3.0)
f = a * a + a
f.backward()
assert a.grad == 7.0

• A primitive test checks the local derivative. A reuse test checks accumulation and ordering together.基本运算测试检查局部导数；复用测试同时检查累加与执行顺序。
• On the reuse example: running p before f gives 1; replacing both multiply updates with assignment gives 3.在复用例子中，先运行 p 再运行 f 会得到 1；把乘法的两条累加改成赋值会得到 3。
• Calling f.backward() again without resetting intermediate gradients gives 20 in this toy engine. Training uses a fresh graph.在这个玩具引擎中，不清空中间梯度就再次调用 f.backward() 会得到 20。训练应使用新图。

Reference comparison: one expression for both engines参考对照：两个引擎运行同一个表达式

def expression(a, b, c):
    d = a * b + c
    e = (d + a).tanh()
    return e * d + b

a=0.7, b=-1.3, c=2.0. Call this function with Value nodes, then with torch tensors.a=0.7, b=-1.3, c=2.0。先传入 Value 节点，再传入 torch 张量。

• The expression reuses d, gives a a second path, and gives b a direct path to the output.表达式复用 d，让 a 多出一条路径，也让 b 直接连到输出。
• Both types implement multiplication, addition and tanh. Python dispatches each operation to that type’s implementation.两种类型都支持乘法、加法和 tanh；Python 把各次运算分派给相应类型的实现。

Reference comparison: run backward and compare every input参考对照：执行反向并比较每个输入

a, b, c = map(Value, (0.7, -1.3, 2.0))
out = expression(a, b, c)
out.backward()

import torch
ts = [torch.tensor(v, dtype=torch.float32,
                   requires_grad=True)
      for v in (0.7, -1.3, 2.0)]
at, bt, ct = ts
outt = expression(at, bt, ct)
outt.backward()

               micro       torch      |diff|
out        -0.269121   -0.269121    5.19e-08
dL/da      -1.263999   -1.263999    2.42e-08
dL/db       1.742557    1.742557    4.19e-08
dL/dc       1.060796    1.060796    5.93e-08

Compare out.data with outt.item(), and each Value.grad with the matching tensor.grad.item().将 out.data 与 outt.item() 比较，再将每个 Value.grad 与对应 tensor.grad.item() 比较。

This example uses Python float versus torch float32. Use tolerances; matching rounded printouts is not exact equality.这里比较 Python float 与 torch float32，应使用容差；打印后相同不代表数值完全相等。