Module 4 · Reading 4.2 · read before S24: GANs

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

The VAE generates by construction (sample from the prior, decode) but pays for it with blur: its loss averages over uncertainty, and averages of sharp things are soft things. In 2014, generative adversarial networks (GANs) proposed a radically different route to the same goal. Instead of writing down a likelihood-based loss that says “make samples like the data,” a GAN learns the loss itself, in the form of a second network trained to tell real from fake.

The idea produced the sharpest generated images the field had seen, dominated image generation for half a decade, and introduced adversarial training to the standard toolkit. It also produced some of the most notoriously unstable training dynamics in deep learning — instability that is not an implementation detail but a direct consequence of the formulation. This section builds a GAN small enough that you can watch both the idea and the pathology happen in real numbers.

How to read this

Two networks, one game

A GAN has two players. The generator \(G\) maps random noise \(z \sim \mathcal{N}(0, I)\) to a fake sample \(G(z)\), like a VAE decoder but with no encoder anywhere. The discriminator \(D\) takes a sample and outputs the probability that it is real. They are trained on opposite objectives, formalized in Goodfellow et al. (NeurIPS 2014) as a minimax game:

\[\min_G \max_D \; \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]\]

\(D\) is a plain binary classifier: maximize log-probability of “real” on data, “fake” on generations. \(G\) is trained to make \(D\) wrong. The elegance is what this buys you: nobody ever writes down what a “good sample” looks like. \(D\) learns the difference between real and generated data, and its gradients tell \(G\) exactly in which direction its samples are detectably fake. As \(G\) improves, \(D\) sharpens its criterion, and the loss \(G\) faces gets harder: a curriculum that upgrades itself.

The paper also proved a clean theoretical property: for a fixed \(G\), the optimal discriminator is \(D^*(x) = p_{\text{data}}(x) / \bigl(p_{\text{data}}(x) + p_g(x)\bigr)\), and at the game’s equilibrium \(p_g = p_{\text{data}}\), where \(D\) outputs 0.5 everywhere and (in our two-term convention below) the discriminator’s loss is \(2 \log 2 \approx 1.386\). Keep that number in mind; you are about to see it appear in a real run.

One practical amendment from the same paper matters for every implementation: early in training, when the discriminator rejects fakes with ease, \(\log(1 - D(G(z)))\) is flat and gives \(G\) almost no gradient; the generator’s loss saturates. The standard fix is the non-saturating objective: instead of minimizing the probability of being caught, \(G\) maximizes \(\log D(G(z))\) (“make D say real”). Same fixed point, much stronger gradient when \(G\) is losing.

Historical note. By Goodfellow’s own account (retold in many interviews), the GAN idea came together during an argument with friends in a Montreal pub in 2014, and the first version was coded the same night. Whatever the exact timeline, the paper appeared at NeurIPS 2014 while Goodfellow was finishing his PhD, and the idea spread unusually fast: within two years Yann LeCun was publicly describing adversarial training as the most interesting machine learning idea of the previous decade.

Why training is a knife-edge

Everything you know about training from earlier chapters assumes one loss and a descent direction on it. A GAN breaks that assumption: there are two losses, each defined relative to the other player’s current parameters. \(G\)’s loss landscape shifts every time \(D\) updates, and vice versa. Simultaneous gradient descent on a game is not guaranteed to converge to the equilibrium — the players can orbit it, overshoot it, or collapse into degenerate strategies.

The most famous degenerate strategy is mode collapse. Nothing in \(G\)’s objective says “cover the whole data distribution.” It says “produce samples \(D\) currently accepts.” If one region of data space fools \(D\) today, the cheapest strategy is to put all the probability mass there. \(D\) eventually notices that this one region is overpopulated with fakes and starts rejecting it, so \(G\) jumps to another region, and the pair can cycle forever, with \(G\) covering one mode at a time instead of all modes at once. The failure is structural: it comes from optimizing a best-response dynamic, not from bad hyperparameters, though hyperparameters decide how badly it bites.

You do not have to take this on faith. Here is a complete GAN on a 1D dataset with exactly two modes: a mixture of Gaussians at -2 and +2. Mode coverage is measurable: just count generated samples near each mode.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Real data: 1D mixture, half N(-2, 0.5^2), half N(+2, 0.5^2)
def real_batch(n):
    side = (torch.rand(n, 1) < 0.5).float() * 4 - 2
    return side + 0.5 * torch.randn(n, 1)

G = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
D = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
ones, zeros = torch.ones(128, 1), torch.zeros(128, 1)

for step in range(2001):
    # --- discriminator step ---
    x_real = real_batch(128)
    x_fake = G(torch.randn(128, 1)).detach()
    d_loss = bce(D(x_real), ones) + bce(D(x_fake), zeros)
    opt_d.zero_grad(); d_loss.backward(); opt_d.step()
    # --- generator step (non-saturating loss) ---
    x_fake = G(torch.randn(128, 1))
    g_loss = bce(D(x_fake), ones)
    opt_g.zero_grad(); g_loss.backward(); opt_g.step()

    if step % 250 == 0:
        with torch.no_grad():
            s = G(torch.randn(4000, 1))
        left = ((s + 2).abs() < 1).float().mean().item()
        right = ((s - 2).abs() < 1).float().mean().item()
        print(f"step {step:4d}  D loss {d_loss.item():.4f}  G loss {g_loss.item():.4f}  "
              f"near -2: {left:.3f}  near +2: {right:.3f}")

Running this prints:

step    0  D loss 1.3810  G loss 0.8556  near -2: 0.000  near +2: 0.000
step  250  D loss 1.0758  G loss 1.0564  near -2: 0.949  near +2: 0.000
step  500  D loss 0.9873  G loss 1.1501  near -2: 0.870  near +2: 0.000
step  750  D loss 1.3852  G loss 0.8334  near -2: 0.223  near +2: 0.331
step 1000  D loss 1.3553  G loss 0.7481  near -2: 0.433  near +2: 0.211
step 1250  D loss 1.3392  G loss 0.7115  near -2: 0.474  near +2: 0.329
step 1500  D loss 1.3392  G loss 0.7200  near -2: 0.468  near +2: 0.377
step 1750  D loss 1.3759  G loss 0.7092  near -2: 0.453  near +2: 0.426
step 2000  D loss 1.3621  G loss 0.7391  near -2: 0.464  near +2: 0.435

This run happens to tell the whole story in nine lines. By step 250 the generator has mode-collapsed: 94.9% of its samples sit at the left mode and none near the right — it found one region that fools D and moved everything there. Notice the discriminator’s loss dropping to 0.99 by step 500: D is winning, precisely because it has caught on that the right mode contains only real data and the left mode is flooded with fakes. That pressure is what pries the generator loose: between steps 500 and 750 mass migrates rightward, and by the end the split is 46.4% / 43.5%, close to what the real data puts in the same windows (about 48% / 48%). The recovery is real but not guaranteed; on harder data the cycle of collapse and escape can repeat indefinitely.

Now look at the loss columns and notice what they do not tell you. At step 250, deep in mode collapse and arguably the worst model of the run, the losses (1.08, 1.06) look unremarkable. In the healthy final stretch, D’s loss hovers around 1.34–1.38, close to the theoretical equilibrium value \(2 \log 2 \approx 1.386\), and G’s near \(\log 2 \approx 0.693\). That proximity is encouraging but not proof: losses near equilibrium values mean D currently cannot separate real from fake, which is also what you would see if D were undertrained or broken. GAN losses are diagnostics of the game state, not of sample quality — which is why we measured mode coverage directly, and why GAN research had to invent separate evaluation metrics (Inception Score, FID) rather than reading quality off the training curves.

Pitfall. Detach G(z) during the discriminator step. Without it, backward constructs and traverses an unnecessary generator graph, deposits generator gradients, and consumes extra memory/compute. In the loop shown here, opt_g.zero_grad() immediately before the generator backward clears those leaked gradients, so they do not contaminate the next step; other loop orderings may accidentally retain them. Detach is still the correct ownership boundary.

Counting modes in 2D

One dimension with two modes is the gentlest possible test. A standard, slightly harder benchmark distribution is a ring of eight Gaussians in 2D, a classic testbed in the GAN literature precisely because mode dropping is so easy to see. A mode counts as “covered” if at least 2% of generated samples land within three standard deviations of its center.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Real data: mixture of 8 Gaussians on a ring of radius 2, std 0.15
angles = torch.arange(8) * (2 * torch.pi / 8)
centers = torch.stack([2 * torch.cos(angles), 2 * torch.sin(angles)], dim=1)  # (8, 2)

def real_batch(n):
    idx = torch.randint(0, 8, (n,))
    return centers[idx] + 0.15 * torch.randn(n, 2)

def modes_covered(samples):
    # a mode counts as covered if >= 2% of samples land within 0.45 of its center
    d = torch.cdist(samples, centers)           # (n, 8)
    nearest = d.argmin(dim=1)
    hit = d.min(dim=1).values < 0.45
    counts = torch.bincount(nearest[hit], minlength=8)
    return (counts.float() / len(samples) >= 0.02).sum().item(), counts.tolist()

G = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 2))
D = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 1))
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
ones, zeros = torch.ones(256, 1), torch.zeros(256, 1)

for step in range(3001):
    x_real = real_batch(256)
    x_fake = G(torch.randn(256, 2)).detach()
    d_loss = bce(D(x_real), ones) + bce(D(x_fake), zeros)
    opt_d.zero_grad(); d_loss.backward(); opt_d.step()

    x_fake = G(torch.randn(256, 2))
    g_loss = bce(D(x_fake), ones)
    opt_g.zero_grad(); g_loss.backward(); opt_g.step()

    if step % 500 == 0:
        with torch.no_grad():
            m, counts = modes_covered(G(torch.randn(4000, 2)))
        print(f"step {step:4d}  D loss {d_loss.item():.4f}  G loss {g_loss.item():.4f}  "
              f"modes covered: {m}/8")

with torch.no_grad():
    m, counts = modes_covered(G(torch.randn(4000, 2)))
print(f"final per-mode sample counts (of 4000): {counts}")

Running this prints:

step    0  D loss 1.3226  G loss 0.6931  modes covered: 0/8
step  500  D loss 1.4362  G loss 0.6343  modes covered: 2/8
step 1000  D loss 1.0848  G loss 0.9441  modes covered: 4/8
step 1500  D loss 1.3097  G loss 0.8107  modes covered: 6/8
step 2000  D loss 1.3044  G loss 0.7653  modes covered: 7/8
step 2500  D loss 1.3096  G loss 0.7395  modes covered: 8/8
step 3000  D loss 1.3032  G loss 0.7528  modes covered: 8/8
final per-mode sample counts (of 4000): [223, 315, 410, 387, 300, 249, 384, 361]

On this run, coverage grows mode by mode (2, 4, 6, 7, then all 8 by step 2500) rather than appearing everywhere at once, which is the best-response dynamic in action: the generator claims territory where the discriminator is currently weakest. But look closer at the final counts. A perfect model would put about 500 samples on each mode; ours ranges from 223 to 410, and the eight counts sum to 2,629 — meaning roughly a third of all samples fell outside every mode’s 3-sigma radius, mostly in the connecting regions between modes. “All modes covered” is a much weaker statement than “distribution matched.” Getting from the former to the latter is where GAN training budgets actually went, and a two-line summary statistic like ours is the kind of check you should build for any generative project before trusting your eyes.

Why distance matters: from JS to Wasserstein

The 2014 paper showed that, at the optimal discriminator, the generator’s objective is (up to constants) the Jensen-Shannon divergence between the data distribution and the generator’s distribution. That sounds like a satisfying theoretical anchor until you ask what the JS divergence does when the two distributions do not overlap. The answer, worked out carefully by Arjovsky and Bottou (“Towards Principled Methods for Training Generative Adversarial Networks,” ICLR 2017), is: nothing. For distributions with disjoint supports, JS is a constant (\(\log 2\)) no matter whether the fake samples are a millimeter from the data or a mile. And disjoint supports are not an edge case: real data concentrates near a low-dimensional manifold, an untrained generator’s output lies on a different one, and two low-dimensional surfaces in a high-dimensional space typically do not intersect. In that regime a well-trained discriminator becomes perfect, and a perfect discriminator hands the generator a gradient of zero: the better \(D\) does its job, the less it teaches \(G\).

The Wasserstein GAN (Arjovsky, Chintala & Bottou, ICML 2017) replaces JS with the Wasserstein-1 distance — informally, the minimum cost of transporting the generator’s probability mass onto the data’s, where moving mass twice as far costs twice as much. Unlike JS, this distance changes smoothly with the gap between the distributions, so its gradient points home from any distance. Practically, the discriminator becomes a critic: no sigmoid, real-valued scores, trained to maximize \(\mathbb{E}[C(\text{real})] - \mathbb{E}[C(\text{fake})]\), but only within the class of functions whose slope never exceeds 1 (1-Lipschitz functions), because without that constraint the maximization diverges. The original paper enforced the constraint crudely, by clipping every weight into a small box; the now-standard gradient penalty (WGAN-GP, Gulrajani et al., NeurIPS 2017) does it softly, adding a term that pushes the critic’s gradient norm toward 1 at points sampled between real and fake data.

All of this is checkable in one controlled experiment, and it is worth doing because the claim (“the discriminator’s gradient vanishes when it wins”) sounds like it should apply to every GAN loss, and it does not. The setup: real data is a cluster at +2; the “generator” is a cloud at a trainable offset mu, so the entire generator update is one number, d loss / d mu. For each distance we train a discriminator to convergence, then measure the size of that gradient under three objectives: the original saturating loss \(\log(1 - D)\), the non-saturating fix \(-\log D\), and a WGAN-GP critic.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Real data: a cluster at +2. "Generator": a cloud at trainable offset mu.
real = 2.0 + 0.25 * torch.randn(512, 1)
eps = 0.25 * torch.randn(512, 1)          # fixed noise for the fake cloud

def make_net():
    return nn.Sequential(nn.Linear(1, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(),
                         nn.Linear(64, 1))

def bce_gradient(mu_val):
    """Train a BCE discriminator to convergence, then measure the generator's
    gradient d/dmu of the non-saturating loss -log D(fake)."""
    torch.manual_seed(0)
    D = make_net()
    opt = torch.optim.Adam(D.parameters(), lr=1e-3)
    bce = nn.BCEWithLogitsLoss()
    fake = mu_val + eps
    for _ in range(500):
        loss = bce(D(real), torch.ones(512, 1)) + bce(D(fake), torch.zeros(512, 1))
        opt.zero_grad(); loss.backward(); opt.step()
    # saturating (original minimax) G loss: minimize log(1 - D(fake))
    mu = torch.tensor(mu_val, requires_grad=True)
    sat = torch.log(1 - torch.sigmoid(D(mu + eps)) + 1e-12).mean()
    sat.backward()
    g_sat = mu.grad.abs().item()
    # non-saturating G loss: minimize -log D(fake)
    mu = torch.tensor(mu_val, requires_grad=True)
    bce(D(mu + eps), torch.ones(512, 1)).backward()
    g_nonsat = mu.grad.abs().item()
    with torch.no_grad():
        p_fake = torch.sigmoid(D(mu_val + eps)).mean().item()
    return p_fake, g_sat, g_nonsat

def wgan_gradient(mu_val):
    """Train a critic with gradient penalty, then measure d/dmu of -C(fake)."""
    torch.manual_seed(0)
    C = make_net()
    opt = torch.optim.Adam(C.parameters(), lr=1e-3)
    fake = mu_val + eps
    for _ in range(500):
        a = torch.rand(512, 1)
        x_hat = (a * real + (1 - a) * fake).requires_grad_(True)
        grad = torch.autograd.grad(C(x_hat).sum(), x_hat, create_graph=True)[0]
        gp = ((grad.norm(2, dim=1) - 1) ** 2).mean()
        loss = C(fake).mean() - C(real).mean() + 10.0 * gp
        opt.zero_grad(); loss.backward(); opt.step()
    mu = torch.tensor(mu_val, requires_grad=True)
    (-C(mu + eps).mean()).backward()
    return mu.grad.abs().item()

print(f"{'mu':>6} {'distance':>9} {'D(fake)':>8} {'|sat grad|':>11} "
      f"{'|non-sat grad|':>15} {'|W grad|':>9}")
for mu_val in [1.5, 0.0, -2.0, -5.0, -8.0]:
    p, g_sat, g_nonsat = bce_gradient(mu_val)
    g_w = wgan_gradient(mu_val)
    print(f"{mu_val:>6.1f} {2.0 - mu_val:>9.1f} {p:>8.4f} {g_sat:>11.6f} "
          f"{g_nonsat:>15.4f} {g_w:>9.4f}")

Running this prints:

    mu  distance  D(fake)  |sat grad|  |non-sat grad|  |W grad|
   1.5       0.5   0.2175    1.738256          5.7076    0.9699
   0.0       2.0   0.0006    0.005676          4.2828    1.0837
  -2.0       4.0   0.0002    0.000913          4.0846    1.0729
  -5.0       7.0   0.0002    0.000324          1.5993    1.2986
  -8.0      10.0   0.0001    0.000137          1.0781    1.4658

Each column tells a different part of the story. The saturating loss behaves exactly as the theory warns: once the clouds separate and D(fake) drops to 0.0006, the generator’s gradient collapses, by four orders of magnitude between overlapping and well-separated clouds. This is the “generator gets nothing” regime, measured. The non-saturating loss genuinely rescues the magnitude (gradients between 1.1 and 5.7 at every distance), which is why it became the default. But look at how arbitrary those magnitudes are: 5.7 at distance 0.5, 4.1 at distance 4, 1.1 at distance 10. The gradient’s size depends on how the trained network happens to extrapolate its logits into empty space, not on anything about the distributions. The WGAN-GP critic’s gradient, by contrast, stays within a factor of 1.5 of exactly 1.0 at every distance — the constrained critic approximates a unit-slope ramp pointing at the data, which is precisely the transport map the Wasserstein distance defines. A steady, distance-meaningful signal at any separation: that is what the reformulation buys, and why WGAN losses correlate with sample quality well enough to be useful monitoring tools.

One honest caveat belongs next to that success. A well-behaved gradient per step does not guarantee a well-behaved game: the two-player dynamics from earlier in this section still apply, the critic must be kept accurately trained (the WGAN-GP paper uses five critic steps per generator step), and on some problems the constrained critic’s slow reshaping introduces its own failure modes. The field’s later convergence analyses (notably Mescheder, Geiger & Nowozin, ICML 2018) found that what stabilizes GANs most reliably is regularizing the discriminator’s gradients near the data — their R1 penalty, which became the workhorse regularizer of the StyleGAN line.

Constraining the critic: spectral normalization

The gradient penalty enforces behavior at sampled points. Spectral normalization instead divides a weight by an estimate of its top singular value. With exact singular values and 1-Lipschitz activations, normalizing every layer gives an upper bound of 1 on the network Lipschitz constant. PyTorch updates the estimate using power iteration, so the implementation is approximate and depends on convergence of that estimate.

import torch
import torch.nn as nn

torch.manual_seed(0)

def real_batch(n):
    side = (torch.rand(n, 1) < 0.5).float() * 4 - 2
    return side + 0.5 * torch.randn(n, 1)

def run(spectral, steps=2000):
    torch.manual_seed(0)
    G = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(),
                      nn.Linear(32, 1))
    layers = [nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(),
              nn.Linear(32, 1)]
    if spectral:
        layers = [nn.utils.spectral_norm(l) if isinstance(l, nn.Linear) else l
                  for l in layers]
    D = nn.Sequential(*layers)
    opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
    opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
    bce = nn.BCEWithLogitsLoss()
    ones, zeros = torch.ones(128, 1), torch.zeros(128, 1)
    for step in range(steps):
        x_real = real_batch(128)
        x_fake = G(torch.randn(128, 1)).detach()
        d_loss = bce(D(x_real), ones) + bce(D(x_fake), zeros)
        opt_d.zero_grad(); d_loss.backward(); opt_d.step()
        x_fake = G(torch.randn(128, 1))
        g_loss = bce(D(x_fake), ones)
        opt_g.zero_grad(); g_loss.backward(); opt_g.step()

    with torch.no_grad():
        s = G(torch.randn(4000, 1))
    left = ((s + 2).abs() < 1).float().mean().item()
    right = ((s - 2).abs() < 1).float().mean().item()
    # top singular value of each linear layer = its Lipschitz constant
    svs = [torch.linalg.svdvals(m.weight.detach())[0].item()
           for m in D.modules() if isinstance(m, nn.Linear)]
    # steepest slope of D over a dense grid of inputs
    x = torch.linspace(-6, 6, 2001).unsqueeze(1).requires_grad_(True)
    grad = torch.autograd.grad(D(x).sum(), x)[0]
    return left, right, svs, grad.abs().max().item()

for spectral in [False, True]:
    left, right, svs, gmax = run(spectral)
    name = "spectral-norm D" if spectral else "plain D        "
    print(f"{name}  near -2: {left:.3f}  near +2: {right:.3f}  "
          f"layer sigma_max {[f'{v:.2f}' for v in svs]}  max |dD/dx| {gmax:.3f}")

Running this prints:

plain D          near -2: 0.471  near +2: 0.434  layer sigma_max ['3.58', '2.03', '0.97']  max |dD/dx| 1.657
spectral-norm D  near -2: 0.445  near +2: 0.463  layer sigma_max ['1.00', '1.00', '1.00']  max |dD/dx| 0.323

The contrast is visible in the run: the plain discriminator’s layers drift to top singular values of 3.58 and 2.03 (steepest observed slope 1.66), while the normalized layers report values rounded to 1.00 and an observed input slope of 0.32. This illustrates the mechanism, but the wrapper’s power-iteration estimate should not be described as an exact proof about every finite-precision forward pass. On this toy, final mode coverage is indistinguishable between the two runs (0.471/0.434 versus 0.445/0.463) — and that null result is the correct reading, not a disappointment: a two-mode problem is too easy to expose stability differences, and the evidence for spectral normalization in the literature is on image-scale GANs.

Evaluating an implicit model without tractable likelihood

Step back and notice a strange absence. A standard GAN is an implicit model: it supports sampling but not tractable pointwise likelihood. VAEs optimize likelihood bounds. Diffusion formulations can provide variational bounds or likelihood estimates, and probability-flow ODE variants can enable density evaluation under additional numerical assumptions; this is formulation-dependent rather than automatic.

The field’s response was to invent sample-based proxies, and two recur in GAN papers. The Inception Score (Salimans et al., NeurIPS 2016) runs generated images through a pretrained ImageNet classifier and rewards samples that are individually confidently classified but collectively diverse — a heuristic with known blind spots, the sharpest being that it never looks at the real data at all. The Fréchet Inception Distance (FID, Heusel et al., NeurIPS 2017) does: it embeds real and generated samples, fits a Gaussian to each feature cloud, and compares those Gaussians. FID zero means the fitted feature Gaussians have the same mean and covariance; it does not prove the underlying image or feature distributions are identical.

FID’s caveats matter in practice. It is biased at small sample counts (the standard convention is 50,000 samples, and numbers computed at different sample sizes are not comparable); it depends on the embedding network, so scores across papers using different feature extractors do not compare; and it compresses two different failures (dropping modes and producing low-quality samples) into one number. Later work (notably Kynkäänniemi et al., NeurIPS 2019) split the number back into precision (are generated samples realistic?) and recall (is all of the data covered?), which maps exactly onto the two failure axes you have now seen: a mode-collapsed generator has high precision and terrible recall; a blurry VAE tends toward the reverse.

Pitfall. Because FID must be estimated from samples, it has run-to-run variance like any other statistic, and small FID differences between checkpoints are routinely inside that variance. The same discipline you will meet in the evaluation section applies: recompute the metric with a different sample draw (or several) before believing an improvement, and be suspicious of any comparison that hinges on a fraction of an FID point.

What became of GANs

For about six years, GANs were image generation. DCGAN (Radford, Metz & Chintala, ICLR 2016) established convolutional architecture guidelines that made training reproducible enough for widespread use. Wasserstein GAN (Arjovsky, Chintala & Bottou, ICML 2017) reframed the objective to give smoother gradients and losses that correlate somewhat with quality. StyleGAN (Karras et al., CVPR 2019) produced face images that most people could not distinguish from photographs, and BigGAN (Brock et al., ICLR 2019) scaled class-conditional generation to ImageNet. Each milestone was, in large part, a fix for the training pathologies you just watched: better architectures, better objectives, better regularization — engineering against the instability inherent in the game.

Then the field moved. Dhariwal and Nichol’s “Diffusion Models Beat GANs on Image Synthesis” (NeurIPS 2021) reported diffusion models overtaking the best GANs on standard image benchmarks, and the flagship generators that followed (Stable Diffusion, DALL-E 2, Imagen and their successors) are diffusion-based, not adversarial. The reasons trace directly back to this section’s demos: diffusion models train with a plain regression loss, no game, so they are stable, and their likelihood-adjacent objective pushes them to cover all modes rather than to satisfy a discriminator.

But “superseded as the flagship” is not “dead.” A GAN generates in a single forward pass, while a naive diffusion sampler needs hundreds; GANs and adversarial losses remain in production wherever that speed matters (real-time super-resolution and image restoration, vocoders for speech synthesis), and adversarial training reappears inside diffusion pipelines as a distillation tool for collapsing many denoising steps into few. The adversarial idea outlived the architecture that made it famous: training a network against a learned critic is now a permanent part of the field’s vocabulary, and mode collapse is the permanent cautionary tale about optimizing against a moving target.

The application pattern that aged best is worth spelling out, because it is bigger than GANs. Very early on, Mirza and Osindero (2014) showed that feeding a label to both networks turns a GAN into a conditional generator: \(G(z, y)\) produces samples of class \(y\), and \(D\) judges pairs. Conditioning is what turned generation from a curiosity into an interface, and its most influential form was image-to-image translation: pix2pix (Isola et al., CVPR 2017) trained a conditional GAN to map one image domain to another (edges to photos, day to night, map to aerial) using paired examples and an adversarial loss on top of a plain L1 reconstruction term, with the adversarial part supplying exactly the sharpness that pure regression averages away. CycleGAN (Zhu et al., ICCV 2017) removed the need for paired data with a cycle-consistency loss (translate to the other domain and back, penalize the round-trip error), opening translation between domains where pairs cannot exist, like photographs and paintings. And SRGAN (Ledig et al., CVPR 2017) applied the same recipe to super-resolution, arguing in the paper that adversarial loss recovers photorealistic texture that MSE-trained models blur — the same MSE-averaging pathology you saw motivate this whole section. In speech, adversarially trained vocoders such as HiFi-GAN (Kong et al., NeurIPS 2020) turn spectrograms into waveforms in real time. The common thread: wherever a cheap per-pixel loss produces plausible-but-soft output, an adversarial term is the established sharpener, a loss component rather than a standalone system, which is how the adversarial idea mostly survives today.

People. Ian Goodfellow did the GAN work as a PhD student in Yoshua Bengio’s group at the Université de Montréal, and is also the first author of the 2016 MIT Press textbook Deep Learning (with Bengio and Aaron Courville) — the optional theory reference on this course’s syllabus. He subsequently worked at Google Brain, OpenAI, and Apple, and his adversarial-examples line of work (small input perturbations that flip a classifier’s prediction) grew into the security subfield you will meet again in the safety section.

Going deeper

The convergence question was never fully closed. Simultaneous gradient descent on a two-player game is a genuinely different dynamical system from descent on a loss, and the theory community spent years mapping how different. Analyses of the game’s local dynamics showed that the interaction between the two players adds a rotational component to the vector field (the players orbit the equilibrium instead of approaching it), and that unregularized GAN training on realistic data models is not locally convergent in general. The same line of work (Mescheder, Geiger & Nowozin, “Which Training Methods for GANs do actually Converge?”, ICML 2018) showed that penalizing the discriminator’s gradient norm on real data (the R1 penalty) restores local convergence, which is why R1, not a Wasserstein loss, is what StyleGAN-class models actually shipped with. The practical residue of all this theory fits in one sentence: modern recipes stabilize the discriminator’s local behavior (R1, spectral norm) and manage the two-timescale balance (separate learning rates for \(G\) and \(D\), proposed alongside FID in the Heusel et al. 2017 paper), rather than trusting any single objective to make the game well-behaved.

Architecture mattered as much as objectives. The StyleGAN series is the clearest case study. Its central idea was to stop feeding noise into the generator’s front door: a mapping network first transforms \(z\) into an intermediate latent \(w\), and \(w\) then modulates the style of every layer of a synthesis network (per-channel scales), with fresh per-pixel noise injected at each resolution for fine texture. The design gave unprecedented control (coarse layers set pose, fine layers set texture), and its latent space proved smooth enough that “GAN inversion” (encoding a real photo into \(w\) and editing it there) became a small subfield of its own. Successive versions diagnosed and fixed their own artifacts (StyleGAN2 removed the telltale “water droplet” blobs by restructuring normalization; StyleGAN3 attacked aliasing that glued texture to pixel coordinates), a rare example of a generative architecture being debugged in public across papers.

The adversarial idea’s afterlife in diffusion. The reason this section still earns its place in a 2026 course is that adversarial training became a component technology. Distillation methods that compress a many-step diffusion sampler into one to four steps commonly add a discriminator to keep the few-step outputs sharp — adversarial diffusion distillation (Sauer et al., 2023, the basis of SDXL-Turbo-style fast samplers) is the direct descendant of everything in this section, with the diffusion model playing the role the dataset used to play. The lesson to carry forward: objectives outlive architectures. The bottleneck idea outlived the autoencoder’s pretraining role; the adversarial loss outlived the GAN’s flagship role; and when you meet a new generative system, asking “what supplies the sharpness, and what supplies the coverage?” will usually decompose it into pieces you already know.

What a GAN’s latent space knows. Like the VAE’s, a trained generator’s latent space supports arithmetic: interpolations between codes decode to smooth image morphs, and directions in latent space correspond to semantic attributes (the DCGAN paper demonstrated vector arithmetic of the “smiling woman - neutral woman + neutral man” variety). Unlike the VAE’s, the space comes with no encoder — finding the code for a given image is itself an optimization problem, which is what GAN-inversion methods solve. If your project needs controllable editing rather than free generation, this difference is the deciding one, and it is a main reason encoder-equipped architectures (VAEs, and diffusion models with their deterministic DDIM inversion) are often easier to build tools on.

Common pitfalls

  • Losses look stable, so you conclude training is going well. Cause: GAN losses measure the game state, not sample quality — our collapsed model at step 250 had unremarkable losses. Fix: monitor samples directly with a task-appropriate statistic (mode counts, FID on images) at regular intervals.
  • Generated samples are all near-identical. Cause: mode collapse — the generator found one region that currently fools the discriminator. Fix: no single reliable one; standard levers are lower learning rates, more discriminator steps per generator step, minibatch-diversity tricks, or switching to a Wasserstein objective. Detect it early with coverage statistics.
  • Generator loss climbs steadily and samples stop improving. Cause: the discriminator has become too strong, and (with the saturating loss especially) the generator’s gradient vanishes. Fix: use the non-saturating loss, reduce discriminator capacity or learning rate, or add label smoothing on the real labels.
  • Training diverges within the first few hundred steps. Cause: learning rates tuned for single-objective training are usually too high for the two-player dynamic; Adam’s default beta1 = 0.9 also adds momentum that destabilizes games. Fix: the DCGAN-era recipe of a small learning rate (about 2e-4) with betas = (0.5, 0.999) remains a good default, and it is what the examples above use.
  • Early training makes no progress at all and the generator loss is flat. Cause: the saturating minimax loss with a discriminator that is already winning — the regime our gradient table measured at 0.0001. Fix: use the non-saturating objective (as every example here does) or a Wasserstein/GP critic; if using the latter, keep the critic well-trained relative to the generator.
  • A checkpoint is declared better because FID improved by a small margin. Cause: FID is a sample-based estimate with run-to-run variance, and small deltas are often noise; sample count also shifts the number systematically. Fix: fix the sample count by convention, recompute across independent sample draws, and only credit differences that clear the observed spread.

Try it yourself

  1. Rerun the 1D GAN with seeds 1 through 4. Does mode collapse happen in every run? Onto which mode, and does every run recover by step 2000? Summarize coverage at steps 250 and 2000 across seeds.
  2. Raise both learning rates from 2e-4 to 2e-3 and rerun. Describe what happens to the loss traces and the coverage numbers — this is the cheapest instability demonstration available.
  3. In the 8-mode experiment, train the discriminator 3 times per generator step. Does coverage develop faster, slower, or fail? Compare final per-mode counts.
  4. Add one line implementing one-sided label smoothing (real labels 0.9 instead of 1.0) in both scripts and compare the final coverage statistics against the originals.
  5. In the gradient-vanishing experiment, reduce the discriminator training from 500 steps to 50 and rebuild the table. How does the saturating-loss column change, and why would an undertrained discriminator hide the pathology? Relate your answer to the folklore advice of keeping D weak.
  6. Wrap the generator in spectral_norm instead of the discriminator and rerun the coverage experiment. Report what happens, then reason about the asymmetry: what does a Lipschitz constraint mean for a network that must map a unit Gaussian onto modes at distance 2, versus for a critic that only scores inputs?

Further reading

  • Goodfellow et al., “Generative Adversarial Networks,” NeurIPS 2014 — the original paper: the minimax objective, the optimal-discriminator theorem, and the non-saturating fix. arxiv.org/abs/1406.2661
  • Arjovsky & Bottou, “Towards Principled Methods for Training Generative Adversarial Networks,” ICLR 2017 — the disjoint-supports analysis behind the vanishing-gradient table in this section. arxiv.org/abs/1701.04862
  • Arjovsky, Chintala & Bottou, “Wasserstein GAN,” ICML 2017 — the earth-mover reformulation and the critic; read alongside Gulrajani et al.’s gradient-penalty follow-up. arxiv.org/abs/1701.07875
  • Gulrajani et al., “Improved Training of Wasserstein GANs” (WGAN-GP), NeurIPS 2017 — replaces weight clipping with the gradient penalty used in our examples. arxiv.org/abs/1704.00028
  • Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR 2018 — the one-line Lipschitz constraint demonstrated above, at image scale. arxiv.org/abs/1802.05957
  • Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS 2017 — introduces both FID and the two-timescale training analysis. arxiv.org/abs/1706.08500
  • Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks” (pix2pix), CVPR 2017 — the adversarial-loss-as-sharpener pattern that outlived standalone GANs. arxiv.org/abs/1611.07004
  • Karras, Laine & Aila, “A Style-Based Generator Architecture for GANs” (StyleGAN), CVPR 2019 — the peak of GAN image synthesis and the source of the style-modulation ideas in Going deeper. arxiv.org/abs/1812.04948
  • Dhariwal & Nichol, “Diffusion Models Beat GANs on Image Synthesis,” NeurIPS 2021 — the changing of the guard, with the comparison methodology spelled out. arxiv.org/abs/2105.05233
  • Dive into Deep Learning, chapter on Generative Adversarial Networks — a runnable course-level treatment complementing this section’s toy experiments. d2l.ai/chapter_generative-adversarial-networks/index.html

Before class you should be able to

  • State the generator’s and discriminator’s objectives and explain what the non-saturating generator loss fixes.
  • Explain mode collapse as a consequence of best-response dynamics, and name a statistic (mode coverage) that detects it when losses look fine.
  • Spot the missing .detach() in a discriminator-step code snippet and describe what it costs, and when leaked generator gradients would survive to the next step.
  • Explain why a well-trained discriminator can hand the generator a vanishing gradient under the original loss, and what the Wasserstein reformulation buys.
  • Say what constraint spectral normalization enforces and at what granularity (per layer, via the top singular value).
  • Explain why GANs have no held-out likelihood, and describe FID at the “compare feature statistics of real and generated samples” level, including one caveat.
  • Summarize what became of GANs: superseded as flagship generators by diffusion, surviving as an adversarial loss component where sharpness or speed matters.

Connects to

  • This session is covered by Quiz 4 (Module 4: S23–S26), given at S27.
  • The project report is due the week of S27, before the recorded presentation. The mode-coverage checks in this reading are a model for the kind of quantitative sanity checks a generative project’s report should include.
  • The S23–S25 readings are one arc; S25’s diffusion reading resolves the trade-off this session sharpens.

← S23 · Autoencoders and VAEs · All readings · S25 · Diffusion Models →