Module 4 · Reading 4.5 + 4.6 · read before S27: Evaluation & Safety

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

Part A — 4.5 Evaluating Models Honestly

Every claim in deep learning ultimately rests on a number: this model scores 86.2, that one 86.4, so we ship the second one. The trouble is that both numbers are random variables. They depend on which examples happened to land in the test set, which seed initialized the weights, which order the minibatches arrived in — and none of those choices have anything to do with which model is better. If you cannot say how big the noise is, you cannot say whether 86.4 beats 86.2, and an uncomfortable amount of published and internal decision-making does not say.

This section is about the discipline of honest measurement: keeping the test set clean, measuring the noise before comparing models, and recognizing the statistical traps that arise the moment you compare many things at once. Nothing here requires new mathematics: just three habits, each demonstrated with a real experiment small enough to run while you read. These habits are graded directly: HW4 (NumPy only) asks you to build bootstrap CIs and a paired model comparison, and your project report is expected to treat its own numbers this way.

How to read this

Test-set discipline and the many faces of leakage

The train/validation/test split encodes a contract: train on the first, make all decisions on the second, and touch the third once, at the end, to report. The test set’s value is exactly the number of decisions that have not been based on it; every peek spends it. Tune your architecture until the test number improves and you have quietly converted the test set into a second validation set: the number it produces now overstates how the model will do on genuinely new data.

Leakage (test information reaching training) has subtler forms than literally training on test rows, and the subtle forms are the ones that survive code review. Fitting your preprocessing (normalization statistics, vocabulary, feature selection) on the full dataset before splitting leaks test statistics into training. Duplicate or near-duplicate examples that land on both sides of the split leak content. Splitting time-series data randomly instead of chronologically lets the model train on the future. In each case the symptom is the same: validation and test numbers that look great and a deployed model that mysteriously does not.

For large language models the problem changes scale and name: benchmark contamination. When the training corpus is a crawl of the internet, and the benchmark’s questions and answers are posted on the internet, the model may have seen the test (verbatim or paraphrased) during pretraining. Model developers run contamination checks (the GPT-4 technical report, for instance, documents n-gram-overlap checks against its evaluation sets), but detection is imperfect, benchmarks are frequently republished and quoted online, and for closed training corpora outsiders cannot audit at all. A benchmark score for an LLM is therefore an upper-bound-flavored claim: it tells you how the model does on those items assuming it has not effectively memorized them, an assumption that degrades as a benchmark ages in public.

Historical note. In 2019, Recht, Roelofs, Schmidt, and Shankar rebuilt the ImageNet and CIFAR-10 test sets from scratch, following the original collection procedures as closely as they could, and re-evaluated dozens of published classifiers on the new sets (“Do ImageNet Classifiers Generalize to ImageNet?”, ICML 2019). The paper reports accuracy drops of roughly 11–14% on the new ImageNet test set and 3–15% on the new CIFAR-10 set. Notably, model rankings were largely preserved (better models stayed better), but the absolute numbers everyone quoted were inflated by years of the community collectively adapting to one fixed test set. Both findings matter: comparisons within a benchmark are more trustworthy than the scores themselves.

Measure the noise before comparing models

Suppose you compare two architectures: a hidden layer of 16 units versus 32. You train each once, one scores higher, and you conclude the winner is better. The experiment below shows why that conclusion is worthless without error bars. Both configurations are trained five times on identical data (the seed changes only the weight initialization and minibatch order) and evaluated on the same fixed test set of 1,000 points.

import torch
import torch.nn as nn

# Fixed dataset (its own seed, independent of training seeds):
# two overlapping Gaussian blobs in 2D
g = torch.Generator().manual_seed(0)
def blobs(n, gen):
    y = (torch.rand(n, generator=gen) < 0.5).long()
    centers = torch.tensor([[-0.8, -0.8], [0.8, 0.8]])
    x = centers[y] + torch.randn(n, 2, generator=gen)
    return x, y

x_tr, y_tr = blobs(2000, g)
x_te, y_te = blobs(1000, g)

def run(hidden, seed, steps=500):
    torch.manual_seed(seed)   # controls init AND minibatch order
    net = nn.Sequential(nn.Linear(2, hidden), nn.ReLU(), nn.Linear(hidden, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-2)
    for step in range(steps):
        idx = torch.randint(0, len(x_tr), (64,))
        loss = nn.functional.cross_entropy(net(x_tr[idx]), y_tr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (net(x_te).argmax(1) == y_te).float().mean().item()

for hidden in [16, 32]:
    accs = torch.tensor([run(hidden, seed) for seed in range(5)])
    print(f"hidden={hidden:2d}  accs: {[f'{a:.3f}' for a in accs.tolist()]}")
    print(f"hidden={hidden:2d}  mean {accs.mean().item():.4f} +/- {accs.std().item():.4f} (std over 5 seeds)")

Running this prints:

hidden=16  accs: ['0.863', '0.862', '0.860', '0.860', '0.864']
hidden=16  mean 0.8618 +/- 0.0018 (std over 5 seeds)
hidden=32  accs: ['0.860', '0.864', '0.861', '0.856', '0.860']
hidden=32  mean 0.8602 +/- 0.0029 (std over 5 seeds)

Look at what a single-run comparison could have concluded. Seed 4: hidden=16 scores 0.864, hidden=32 scores 0.860 (“smaller is better”). Seed 1: 0.862 versus 0.864 (“bigger is better”). Averaged over five seeds, the doubled network comes out 0.16 percentage points worse (0.8602 vs. 0.8618), a difference comfortably inside the seed-to-seed spread of either configuration. The honest summary is that these two configurations are indistinguishable on this task at this sample size — a conclusion no single pair of runs could have delivered, and a conclusion that is itself useful: it tells you to stop tuning width and spend your budget elsewhere. In published work the same discipline matters more, not less; Dodge et al. (EMNLP 2019) showed that reporting single best runs without variance can reverse the apparent ranking of NLP models entirely.

Error bars from the test set: the bootstrap

Seed variance is only one noise source. The bootstrap estimates test-sample uncertainty by resampling the appropriate independent unit with replacement and recomputing the statistic. It works for many metrics, but not blindly for every dependency structure: paired predictions must be resampled together, grouped/patient data at the group level, time series with block-aware methods, and corpus metrics such as BLEU with their documented resampling unit. Check the estimand and exchangeability assumptions before making it the default.

import torch
import torch.nn as nn

# Same data and model as the seed-variance experiment; one trained model
g = torch.Generator().manual_seed(0)
def blobs(n, gen):
    y = (torch.rand(n, generator=gen) < 0.5).long()
    centers = torch.tensor([[-0.8, -0.8], [0.8, 0.8]])
    x = centers[y] + torch.randn(n, 2, generator=gen)
    return x, y

x_tr, y_tr = blobs(2000, g)
x_te, y_te = blobs(1000, g)

torch.manual_seed(0)
net = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
opt = torch.optim.Adam(net.parameters(), lr=1e-2)
for step in range(500):
    idx = torch.randint(0, len(x_tr), (64,))
    loss = nn.functional.cross_entropy(net(x_tr[idx]), y_tr[idx])
    opt.zero_grad(); loss.backward(); opt.step()

with torch.no_grad():
    correct = (net(x_te).argmax(1) == y_te).float()   # 0/1 per test example

def bootstrap_ci(correct, n_boot=10000, gen=None):
    n = len(correct)
    idx = torch.randint(0, n, (n_boot, n), generator=gen)
    accs = correct[idx].mean(dim=1)                   # one accuracy per resample
    lo = torch.quantile(accs, 0.025).item()
    hi = torch.quantile(accs, 0.975).item()
    return correct.mean().item(), lo, hi

gb = torch.Generator().manual_seed(1)
acc, lo, hi = bootstrap_ci(correct, gen=gb)
print(f"test n=1000: acc {acc:.4f}  95% bootstrap CI [{lo:.4f}, {hi:.4f}]  "
      f"width {hi-lo:.4f}")
acc, lo, hi = bootstrap_ci(correct[:200], gen=gb)
print(f"test n= 200: acc {acc:.4f}  95% bootstrap CI [{lo:.4f}, {hi:.4f}]  "
      f"width {hi-lo:.4f}")

Running this prints:

test n=1000: acc 0.8630  95% bootstrap CI [0.8410, 0.8840]  width 0.0430
test n= 200: acc 0.8900  95% bootstrap CI [0.8450, 0.9300]  width 0.0850

Two lessons sit in those two lines. A single-model interval spanning 4.3 points shows substantial uncertainty in that model’s absolute accuracy, but it should not be compared literally with the 0.16-point model difference: the relevant interval for the difference is a paired bootstrap over both models’ predictions on the same items. Small test sets still widen uncertainty and invite flattering flukes; use the paired estimand for model comparison.

Pitfall. The bootstrap CI above quantifies test-set sampling noise for one trained model; it says nothing about training noise, and the two do not substitute for each other. A complete comparison needs both: multiple seeds (does the difference survive retraining?) and a paired test-set analysis (does it survive resampling the evaluation data?). Reporting mean +/- std over 5 seeds on a 200-item test set, or a beautiful bootstrap CI on a single run, each tells only half the story — and the half you omitted is usually the one that kills your conclusion.

How big a difference can your test set even see?

The bootstrap told you how uncertain a measured number is. The mirror-image question is the one to ask before running an experiment: if model B really is better than model A by some margin, what is the probability my test set will detect it? That probability is called statistical power, and it depends on three things: the true effect size, the test-set size, and your significance threshold. Underpowered evaluations are not merely inconclusive; they are actively misleading, because the differences they do flag are inflated flukes (the only way a noisy measurement clears a high bar is by getting lucky upward — the winner’s-curse mechanism you are about to meet again in sweeps).

Power calculations for accuracy comparisons have textbook formulas, but simulation is more honest about what you actually do with a benchmark, and it takes eight lines: draw both models’ scores at their true accuracies, run the comparison test, repeat a few thousand times, count detections.

import torch

torch.manual_seed(0)

def detect_rate(p_a, p_b, n, trials=3000, z_crit=1.645):
    """Fraction of experiments where a two-proportion z-test finds B > A
    significant at alpha = 0.05, with n test examples per model."""
    acc_a = torch.distributions.Binomial(n, torch.tensor(p_a)).sample((trials,)) / n
    acc_b = torch.distributions.Binomial(n, torch.tensor(p_b)).sample((trials,)) / n
    pool = (acc_a + acc_b) / 2
    se = (2 * pool * (1 - pool) / n).sqrt()
    z = (acc_b - acc_a) / se
    return (z > z_crit).float().mean().item()

print("power to detect a true accuracy gain (alpha = 0.05, one-sided)")
print(f"{'n test':>8} {'+0.5 pt':>8} {'+1 pt':>7} {'+2 pt':>7} {'+5 pt':>7} "
      f"{'false pos (+0)':>15}")
for n in [200, 500, 1000, 5000, 20000]:
    row = [detect_rate(0.85, 0.85 + d, n) for d in [0.005, 0.01, 0.02, 0.05]]
    fp = detect_rate(0.85, 0.85, n)
    print(f"{n:>8} {row[0]:>8.3f} {row[1]:>7.3f} {row[2]:>7.3f} {row[3]:>7.3f} "
          f"{fp:>15.3f}")

Running this prints:

power to detect a true accuracy gain (alpha = 0.05, one-sided)
  n test  +0.5 pt   +1 pt   +2 pt   +5 pt  false pos (+0)
     200    0.063   0.086   0.136   0.465           0.051
     500    0.067   0.114   0.227   0.771           0.055
    1000    0.086   0.159   0.352   0.957           0.048
    5000    0.176   0.412   0.890   1.000           0.051
   20000    0.422   0.886   1.000   1.000           0.052

This table is deliberately a conservative independent-binomial design, useful for intuition but not a description of the usual same-item model comparison. Real evaluations should exploit paired correctness and item difficulty with McNemar-style tests or paired bootstrap; power can be substantially higher. The \(1/\sqrt{n}\) scaling remains useful, but quote the table only with its independent-design assumption.

Two smaller observations round out the table. The false-positive column hovers at 0.048–0.055 — right at the nominal 0.05, which is the sanity check that the test is calibrated: a procedure that promises a 5% false-alarm rate under the null should deliver about 5% in simulation, and this one does. And the +5-point column shows what adequate power looks like: 96% detection at n = 1000. If your project’s improvements are of that size, ordinary benchmarks serve you fine; if you are hunting one-point gains, you now know the price of being able to see them. This style of reasoning has its own literature in NLP: Card et al. (“With Little Power Comes Great Responsibility,” EMNLP 2020) surveyed common evaluation setups and reported that many typical NLP experimental designs are underpowered for the effect sizes researchers actually chase.

Pitfall. “The difference was not statistically significant” does not mean “there is no difference” — at n = 1000, real one-point gains go undetected 84% of the time in our table. Absence of evidence at low power is close to no information at all. The honest phrasing, which your HW4 and project reports should use, is: “our evaluation could not distinguish the two at this sample size,” ideally with the minimal detectable effect stated. Conversely, do not let a significant result from a small test set impress you: at low power, significant findings are systematically exaggerated.

Comparing many things: the multiple-comparisons trap

Now the trap that catches even people who do everything above correctly. Hyperparameter search means comparing many configurations, and the more numbers you compare, the more extreme the maximum of them will be by luck alone. The experiment below trains twenty copies of the identical configuration, differing only in seed, and then does what a leaderboard does: crowns the best.

import torch
import torch.nn as nn

# 20 runs of the SAME config (hidden=16), differing only in seed
g = torch.Generator().manual_seed(0)
def blobs(n, gen):
    y = (torch.rand(n, generator=gen) < 0.5).long()
    centers = torch.tensor([[-0.8, -0.8], [0.8, 0.8]])
    x = centers[y] + torch.randn(n, 2, generator=gen)
    return x, y

x_tr, y_tr = blobs(2000, g)
x_te, y_te = blobs(1000, g)

def run(seed):
    torch.manual_seed(seed)
    net = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-2)
    for step in range(500):
        idx = torch.randint(0, len(x_tr), (64,))
        loss = nn.functional.cross_entropy(net(x_tr[idx]), y_tr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (net(x_te).argmax(1) == y_te).float().mean().item()

accs = torch.tensor([run(seed) for seed in range(20)])
best, worst = accs.max().item(), accs.min().item()
print(f"20 identical configs, different seeds:")
print(f"  mean {accs.mean().item():.4f}  std {accs.std().item():.4f}")
print(f"  best seed  {accs.argmax().item():2d}: {best:.4f}")
print(f"  worst seed {accs.argmin().item():2d}: {worst:.4f}")
print(f"  best - worst = {best - worst:.4f}  "
      f"({(best - worst) * 1000:.0f} test examples out of 1000)")

Running this prints:

20 identical configs, different seeds:
  mean 0.8606  std 0.0020
  best seed   4: 0.8640
  worst seed 13: 0.8570
  best - worst = 0.0070  (7 test examples out of 1000)

If these twenty runs had been twenty different hyperparameter settings, an honest-looking report would announce that the winning setting beats the losing one by 0.7 points (a margin larger than many published improvements) when by construction every setting is identical and the gap is seven test examples’ worth of luck. Even a modern-sounding note like “seed 3407 worked best for us” is this trap wearing a lab coat (a 2021 paper by Picard, provocatively titled “torch.manual_seed(3407) is all you need,” scanned thousands of seeds on CIFAR-10 to document how wide such spreads can be). The defenses are procedural, not mathematical: decide the comparison before running it; when you sweep, report the whole sweep, not the max; confirm any “winner” with fresh seeds it was not selected on; and treat any margin smaller than your measured identical-config spread as silence, not signal. This is the same experimental hygiene introduced in experiments and reproducibility, now with the failure mode quantified.

Making the correction: Bonferroni and friends

The procedural defenses have a statistical counterpart. If the \(k\) null tests are independent and each has false-positive rate \(\alpha\), the family-wise rate is \(1 - (1 - \alpha)^k\); dependence changes that exact expression. Bonferroni tests each at \(\alpha/k\) and controls the family-wise rate at most \(\alpha\) without requiring independence.

import torch

torch.manual_seed(0)

# A hyperparameter sweep where NOTHING works: k candidate configs, every one
# with the same true accuracy as the baseline (0.86). Any "significant win"
# is a false positive by construction.
def sweep_experiment(k, n=1000, trials=2000, alpha=0.05):
    p = 0.86
    base = torch.distributions.Binomial(n, torch.tensor(p)).sample((trials, 1)) / n
    cand = torch.distributions.Binomial(n, torch.tensor(p)).sample((trials, k)) / n
    pool = (base + cand) / 2
    se = (2 * pool * (1 - pool) / n).sqrt()
    z = (cand - base) / se
    z_raw = torch.tensor(1.645)                       # one-sided alpha = 0.05
    # Bonferroni: test each of the k comparisons at alpha / k
    z_bonf = torch.distributions.Normal(0., 1.).icdf(torch.tensor(1 - alpha / k))
    fwer_raw = (z > z_raw).any(dim=1).float().mean().item()
    fwer_bonf = (z > z_bonf).any(dim=1).float().mean().item()
    n_sig = (z > z_raw).float().sum(dim=1).mean().item()
    return fwer_raw, fwer_bonf, n_sig

print("k identical configs vs baseline, n = 1000 test examples, alpha = 0.05")
print(f"{'k':>4} {'P(any raw win)':>15} {'avg raw wins':>13} {'P(any Bonf win)':>16}")
for k in [1, 5, 20, 100]:
    raw, bonf, avg = sweep_experiment(k)
    print(f"{k:>4} {raw:>15.3f} {avg:>13.2f} {bonf:>16.3f}")

Running this prints:

k identical configs vs baseline, n = 1000 test examples, alpha = 0.05
   k  P(any raw win)  avg raw wins  P(any Bonf win)
   1           0.060          0.06            0.060
   5           0.197          0.28            0.044
  20           0.345          0.99            0.034
 100           0.575          5.33            0.031

With a single comparison, the uncorrected test behaves near its nominal rate; large sweeps generate false winners. Bonferroni trades power for family-wise control. An independent, pre-specified confirmation on fresh data is a new confirmatory test separated from the exploratory selection; it is not “Bonferroni with \(k=1\)” applied retroactively to the sweep. Keep the exploratory and confirmatory datasets/claims distinct.

What LLM benchmark numbers do and do not mean

Everything above compounds when the subject is a large language model. An LLM benchmark score (MMLU, HumanEval, and their successors) inherits all the usual noise sources, then adds several of its own. Contamination, discussed earlier, inflates scores in ways that are hard to bound. Prompt sensitivity means the score depends on formatting choices: answer-option ordering, few-shot example selection, instruction phrasing; differences between two labs’ “same” evaluation can exceed the differences between their models. Benchmark aging means popular benchmarks saturate and stop discriminating at the top, while their public familiarity keeps growing. And Goodhart’s law (when a measure becomes a target, it ceases to be a good measure) operates at full strength, because benchmark scores drive releases, valuations, and headlines.

None of this makes benchmarks useless; it makes them instruments with error bars and failure modes, like any instrument. A defensible reading of “Model A scores 86.2 on MMLU, Model B scores 85.9” is: on this particular item set, under one particular prompting setup, with unknowable contamination for both models, A and B are within noise of each other — treat them as equivalent until a preregistered evaluation on data neither has seen says otherwise. Efforts like HELM (Liang et al., 2022) push in this direction by evaluating many models under one transparent harness, across many scenarios and metrics rather than a single number. For your own work, the transferable rule is smaller: never report a benchmark delta without an estimate of the noise floor; you now have the tools to produce one in a dozen lines.

Two documented case studies make the abstractions concrete. On contamination: a 2024 study by Zhang and colleagues (“A Careful Examination of Large Language Model Performance on Grade School Arithmetic”) commissioned human writers to build GSM1k, a from-scratch clone of the popular GSM8k math benchmark matched in style and difficulty, then re-evaluated a large slate of models. The paper reports accuracy drops of up to 13 points on the new items, with some open-model families overfit across nearly every size — while several frontier families showed little to no gap. Notably, the authors argue the overfitting is not all literal test-set leakage; collecting training data similar to a famous benchmark, or selecting checkpoints by benchmark score, produces the same signature. This is the Recht experiment of the callout above, replayed for LLMs, with the same moral: a benchmark ages the moment the field starts optimizing toward it. On benchmark quality: the “Are We Done with MMLU?” study (2024) manually re-annotated a sample of MMLU questions and estimated that around 6.5% contain ground-truth errors (with some subject areas far worse), meaning that for top models, a nontrivial slice of the remaining “headroom” on the benchmark is disagreement with wrong answer keys, not missing capability. Saturation completes the picture: once leading models cluster within a few points of each other (and of the error ceiling), a benchmark stops discriminating, which is what drove successors like MMLU-Pro with harder items and more answer options.

When the judge is also a model

A growing share of evaluation cannot be scored by string match at all: open-ended generation, dialogue, code explanations, summaries. The scalable answer (using a strong LLM to grade another LLM’s output, LLM-as-judge) went from improvisation to standard methodology with Zheng and colleagues’ MT-Bench study (NeurIPS 2023), and you should know both halves of its result. The encouraging half: the paper reports that a GPT-4-class judge agreed with human preference judgments over 80% of the time on their benchmarks (comparable to human-human agreement), which is why the method is everywhere from research papers to product dashboards. The cautionary half is a catalog of measured biases. Position bias: when shown two answers, judges favor one position (swapping the order changes verdicts), so serious pipelines score both orderings and count only consistent wins. Verbosity bias: longer answers are favored beyond their content — a judge that can be gamed by padding. Self-enhancement bias: models tend to prefer outputs in their own style, including their own, so a model should not sit on its own jury. None of these is fatal; all of them are systematic, meaning they do not average out with more data the way random noise does.

Treat a judge model, in other words, as a measurement instrument with its own error model — the same stance this section takes toward everything else. That means calibrating it against a sample of human labels for your specific task before trusting it at scale; fixing the judge’s identity, version, and prompt (a judge model update silently redefines your metric); and remembering that judge scores inherit all the earlier problems too: an evaluation set the judge’s training data contains is contaminated at one remove. For a course project using judge-based scoring, the minimum defensible setup is: a written rubric, randomized answer order, a held-out human-labeled calibration sample with the agreement rate reported, and error bars from the bootstrap as for any other metric.

People. Percy Liang, a Stanford professor and director of its Center for Research on Foundation Models, has been one of the most persistent voices for transparent evaluation. His group built HELM (“Holistic Evaluation of Language Models,” 2022), which evaluates models across dozens of scenarios and reports accuracy alongside calibration, robustness, fairness, and efficiency — an explicit rejection of the single-leaderboard-number culture. The same group’s earlier SQuAD benchmark (2016) helped create that culture, which lends the later work a certain earned authority.

Going deeper (Part A)

Paired tests are the free lunch of model comparison. Almost every comparison you will run in practice evaluates two models on the same test items, and that pairing is statistical gold: much of the noise (item difficulty) is shared, and a paired analysis subtracts it. For classification, the classical tool is McNemar’s test, which looks only at the disagreement cells (items A got right and B got wrong, and vice versa) and asks whether their ratio departs from 50/50; items both models get right or both get wrong carry no information about the difference and are correctly ignored. The paired bootstrap from the exercises is the same idea without the formula. The practical effect is substantial: paired designs can detect differences that the independent-samples power table above says are invisible, which is one more reason to evaluate all candidate models on one fixed, versioned test set rather than on “equivalent” splits.

Confidence intervals beat verdicts. A significance test returns a bit; an interval returns the bit and the magnitude and the uncertainty. “B beats A, p < 0.05” and “B improves accuracy by 2.1 points, 95% CI [0.4, 3.8]” are answers to the same experiment, and the second is strictly more useful — it shows the improvement could plausibly be half a point, which may not justify B’s extra cost. Modern reporting guidance across empirical fields has been shifting from p-values toward effect sizes with intervals, and everything in this section (bootstrap, seed spreads) already produces intervals natively. When a deadline forces one number, report the interval’s lower bound for your improvement (the conservative claim) rather than the point estimate.

The reused-test-set problem has real mathematics. Adaptive-data-analysis bounds depend on the query class, desired accuracy/failure probability, analyst adaptivity, and mechanism. Some statistical-query results yield quadratic-in-\(n\) query regimes under specified error assumptions; \(n^2\) is not a universal lifetime for every holdout. The transferable lesson is that adaptive reuse spends statistical validity and specialized mechanisms can account for it.

Evaluation infrastructure is a research artifact now. Two habits of the current era are worth adopting wherever you land. First, harnesses over scripts: shared, versioned evaluation code (HELM; EleutherAI’s lm-evaluation-harness, which standardizes prompting and scoring across hundreds of benchmarks) exists precisely because prompt-formatting differences between two labs’ scripts can exceed model differences — citing the harness version is part of citing the result. Second, evaluation cards over bare numbers: stating what was measured, on which split, with which prompts, at which date, under which contamination checks. The model cards of the next section extend exactly this discipline from performance claims to safety claims.

Common pitfalls (Part A)

  • Test accuracy is excellent; production performance is not. Cause: leakage — preprocessing fit before the split, duplicates across splits, or repeated tuning against the test set. Fix: split first, fit everything on train only, deduplicate across splits, and log how many times the test set has been used.
  • A config change “improves” accuracy by half a point and you ship it. Cause: no noise estimate; typical seed spread and test-set CI are each larger than the improvement. Fix: rerun both configs across several seeds and require the gap to clear the identical-config spread and a paired bootstrap CI.
  • The best setting from a large sweep underperforms when rerun. Cause: winner’s curse — the max over many noisy numbers is biased upward. Fix: re-evaluate the selected config with fresh seeds before believing (or reporting) its number.
  • Two papers report very different scores for the same LLM on the same benchmark. Cause: prompt formatting, few-shot choices, and scoring differences — not model differences. Fix: compare only numbers produced by the same harness, and cite the harness, not just the benchmark.
  • A sweep produces a “significant” winner, but the finding does not replicate. Cause: uncorrected multiple comparisons — at 20 configurations, one sweep in three crowns a false winner at nominal alpha. Fix: Bonferroni (or Holm/FDR) across the sweep, or treat the sweep as exploratory and confirm the winner on fresh seeds and data.
  • An LLM-judge evaluation prefers your new model, and the improvement vanishes under human review. Cause: judge biases (verbosity, position, or style similarity to the judge) rewarded changes that humans do not value. Fix: randomize positions, calibrate the judge against a human-labeled sample, and report the judge’s agreement rate alongside its scores.

Try it yourself (Part A)

  1. Extend the seed-variance experiment to 20 seeds per configuration and check whether the hidden=16 vs. hidden=32 ordering from 5 seeds holds. Does the gap grow, shrink, or change sign?
  2. Implement a paired bootstrap: train both configurations once, keep both correctness vectors, resample indices once per iteration, and compute the CI of the difference in accuracy. Does the interval contain zero?
  3. Simulate contamination: move 100 test examples into the training set (keeping them in the test set too), retrain, and measure the test accuracy change. This is the mechanism behind inflated benchmark scores, at laptop scale.
  4. Using the 20-run experiment, compute how often the best of \(k\) identical configs beats the worst by more than 1 point, for \(k\) in 2, 5, 10, 20 (resample subsets from your 20 runs). At what \(k\) does a 1-point “improvement” become unremarkable?
  5. Extend the power simulation to a paired design: draw per-item correctness for both models with a shared item-difficulty component (e.g., both models correct on the same “easy” 70% of items, differing only on the rest), and detect with McNemar-style disagreement counts. How much does pairing improve the power at n = 1000 for a +2-point effect?
  6. In the Bonferroni simulation, give one of the \(k\) candidates a real +3-point improvement. Measure how often it is the config that gets crowned (a) uncorrected, (b) with Bonferroni. What fraction of the corrected sweeps still find it — that is, what did the correction cost you in power?

Further reading (Part A)

  • Recht, Roelofs, Schmidt & Shankar, “Do ImageNet Classifiers Generalize to ImageNet?”, ICML 2019 — the rebuilt-test-set experiment: inflated absolute scores, largely preserved rankings. arxiv.org/abs/1902.10811
  • Dodge et al., “Show Your Work: Improved Reporting of Experimental Results,” EMNLP 2019 — how single-best-run reporting reverses model rankings, with concrete reporting recommendations. arxiv.org/abs/1909.03004
  • Card et al., “With Little Power Comes Great Responsibility,” EMNLP 2020 — power analysis for NLP evaluations; the field-scale version of our power table. arxiv.org/abs/2010.06595
  • Liang et al., “Holistic Evaluation of Language Models” (HELM), 2022 — many models, many scenarios, many metrics, one transparent harness. arxiv.org/abs/2211.09110
  • Hendrycks et al., “Measuring Massive Multitask Language Understanding” (MMLU), ICLR 2021 — the benchmark whose lifecycle (adoption, saturation, error audits) this section uses as a case study. arxiv.org/abs/2009.03300
  • Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (GSM1k), 2024 — the from-scratch benchmark clone that measured overfitting to GSM8k. arxiv.org/abs/2405.00332
  • Gema et al., “Are We Done with MMLU?”, 2024 — manual re-annotation finding ground-truth errors in a widely used benchmark. arxiv.org/abs/2406.04127
  • Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023 — the standard reference for judge agreement rates and judge biases. arxiv.org/abs/2306.05685
  • Picard, “torch.manual_seed(3407) is all you need,” 2021 — a short, pointed study of seed variance. arxiv.org/abs/2109.08203
  • d2l.ai’s discussion of model selection and generalization — a course-level treatment of the split discipline underlying everything here. d2l.ai/chapter_linear-regression/generalization.html

Part B — 4.6 Safety, Bias, and Responsible Release

This section is not a detour from engineering into ethics; it is engineering. A trained model is an optimizer’s answer to the question “what predicts the labels in this dataset?” — and the optimizer does not know which regularities in the data are the task and which are accidents of collection. When an accident predicts the labels, the model will use it, the test-set numbers from the last section will look fine, and the failure will surface only where it costs the most: after deployment, on the people the accident points at.

The good news is that this failure mode is mechanical, which means it can be demonstrated, measured, and partially engineered away, all of which we will do below with a classifier small enough to inspect completely. The rest of the section covers the infrastructure that has grown around this reality: documentation standards for datasets and models, evaluation aimed at finding failures rather than confirming successes, and the decisions that make up a responsible release.

How bias enters: shortcuts in the loss landscape

Start with the mechanism, stripped of everything but arithmetic. Imagine a screening model where the label truly depends on a continuous “skill” signal, and the data also contains a binary attribute s — think of it as any demographic or demographic-correlated feature. In the historical training data, s happens to agree with the label 90% of the time. The attribute causes nothing; it is merely correlated, the residue of whatever process generated the historical labels. At deployment, the correlation is gone (s agrees with the label only 50% of the time, no better than a coin). What does a perfectly standard classifier do with this?

import torch
import torch.nn as nn

torch.manual_seed(0)

def make_data(n, p_match):
    """Label depends on a real-valued 'skill' signal. A binary attribute s
    matches the label with probability p_match (spurious correlation)."""
    skill = torch.randn(n)
    y = (skill + 0.7 * torch.randn(n) > 0).float()      # noisy function of skill
    match = (torch.rand(n) < p_match).float()
    s = match * y + (1 - match) * (1 - y)               # s = y w.p. p_match
    x = torch.stack([skill, s], dim=1)
    return x, y

x_tr, y_tr = make_data(4000, p_match=0.9)   # strong correlation in training
x_te, y_te = make_data(4000, p_match=0.5)   # correlation absent at deployment

model = nn.Linear(2, 1)
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for step in range(2000):
    loss = nn.functional.binary_cross_entropy_with_logits(model(x_tr).squeeze(1), y_tr)
    opt.zero_grad(); loss.backward(); opt.step()

w = model.weight.detach().squeeze()
print(f"learned weights: skill {w[0].item():.4f}   attribute s {w[1].item():.4f}")

def acc(x, y):
    with torch.no_grad():
        return ((model(x).squeeze(1) > 0).float() == y).float().mean().item()

print(f"train accuracy (correlation 0.9): {acc(x_tr, y_tr):.4f}")
print(f"test accuracy (correlation 0.5):  {acc(x_te, y_te):.4f}")
agree = x_te[:, 1] == y_te
print(f"  test acc when s agrees with y:    {acc(x_te[agree], y_te[agree]):.4f}")
print(f"  test acc when s disagrees with y: {acc(x_te[~agree], y_te[~agree]):.4f}")

Running this prints:

learned weights: skill 2.3325   attribute s 4.3128
train accuracy (correlation 0.9): 0.9220
test accuracy (correlation 0.5):  0.6725
  test acc when s agrees with y:    0.9879
  test acc when s disagrees with y: 0.3627

Read the weights first: the model put nearly twice the weight on the meaningless attribute (4.31) as on the real signal (2.33). It is not malfunctioning: in the training distribution, s was the better feature, and gradient descent has no other notion of “better.” The aggregate numbers then tell the standard shortcut story: 92% on training data, collapsing to 67% when the correlation breaks. But the subgroup split is the line that matters. For people whose attribute happens to agree with their label, accuracy is 98.8%. For those whose attribute disagrees, it is 36.3% — worse than flipping a coin, because the model actively holds their attribute against them. An aggregate metric averages these into a mediocre-but-unalarming 67%; the harm lives entirely in the split. This is why per-group evaluation is not an optional refinement: bias in a model is often invisible in exactly the summary statistics that standard evaluation reports. Geirhos et al. (2020) catalog this pattern, “shortcut learning,” across vision and language; our two-feature version is the smallest member of a large family.

The family’s documented members are worth knowing because they show how ordinary the mechanism is. In the LIME interpretability paper’s famous demonstration (Ribeiro, Singh & Guestrin, KDD 2016), a husky-versus-wolf classifier turned out to be keying on snow in the background: the wolves’ photos were wintry, so snow predicted “wolf” better than the animal did. During the COVID-19 pandemic, a study by DeGrave, Janizek, and Lee (Nature Machine Intelligence, 2021) audited models that claimed to detect COVID from chest X-rays and reported that many were exploiting acquisition artifacts (differences in text markers, positioning, and processing between the hospitals that supplied positive and negative images) rather than lung pathology, meaning their published accuracy measured which hospital took the picture. In each case, the model behaved exactly like our two-feature classifier: it found the cheapest predictive signal in the training distribution, and the signal happened not to be the task. Nothing about demographic attributes is special to the mechanism; what is special is that when the shortcut is demographic (or correlated with demographics, as hospital, zip code, and camera type often are), the failure lands on people, unevenly, which is what promotes an accuracy bug to a fairness harm.

Notice also what this framing does to the word “bias.” In public discussion, model bias is often narrated as a moral failing of the people who built the system. The engineering picture is less comfortable and more actionable: bias is the default outcome of empirical risk minimization on convenient data, produced by exactly the optimization working as designed. No malice is required, and no good intention prevents it — only measurement does. That is why this section keeps reducing ethics-sounding questions to tables you can compute: the tables are where the problem is visible, and therefore where the fix can be verified.

Two mitigations, measured

The mechanism suggests two obvious interventions, and both can be tested in the same sandbox. First, feature removal: do not give the model the attribute. Second, reweighting: keep the feature but reweight the training examples so that, effectively, s and y are statistically independent in the training signal; the rare cells (attribute disagrees with label) get upweighted until the shortcut carries no information.

import torch
import torch.nn as nn

torch.manual_seed(0)

def make_data(n, p_match):
    skill = torch.randn(n)
    y = (skill + 0.7 * torch.randn(n) > 0).float()
    match = (torch.rand(n) < p_match).float()
    s = match * y + (1 - match) * (1 - y)
    x = torch.stack([skill, s], dim=1)
    return x, y

x_tr, y_tr = make_data(4000, p_match=0.9)
x_te, y_te = make_data(4000, p_match=0.5)

def train_eval(name, features, weights=None):
    torch.manual_seed(0)
    model = nn.Linear(features.shape[1], 1)
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    for step in range(2000):
        logits = model(features).squeeze(1)
        loss = nn.functional.binary_cross_entropy_with_logits(
            logits, y_tr, weight=weights)
        opt.zero_grad(); loss.backward(); opt.step()
    x_eval = x_te if features.shape[1] == 2 else x_te[:, :1]
    with torch.no_grad():
        pred = (model(x_eval).squeeze(1) > 0).float()
    agree = x_te[:, 1] == y_te
    overall = (pred == y_te).float().mean().item()
    a = (pred[agree] == y_te[agree]).float().mean().item()
    d = (pred[~agree] == y_te[~agree]).float().mean().item()
    print(f"{name:22}  test acc {overall:.4f}   s==y {a:.4f}   s!=y {d:.4f}   "
          f"gap {a - d:+.4f}")

# Baseline: both features, unweighted (the shortcut learner)
train_eval("baseline (skill + s)", x_tr)

# Fix 1: drop the attribute entirely
train_eval("drop s", x_tr[:, :1])

# Fix 2: keep s, but reweight the four (y, s) cells so s carries no signal
wts = torch.zeros(len(y_tr))
for yv in [0.0, 1.0]:
    for sv in [0.0, 1.0]:
        cell = (y_tr == yv) & (x_tr[:, 1] == sv)
        wts[cell] = len(y_tr) / (4 * cell.sum())     # equalize cell mass
train_eval("reweight (y, s) cells", x_tr, weights=wts)

Running this prints:

baseline (skill + s)    test acc 0.6725   s==y 0.9879   s!=y 0.3627   gap +0.6252
drop s                  test acc 0.8073   s==y 0.8022   s!=y 0.8122   gap -0.0100
reweight (y, s) cells   test acc 0.8073   s==y 0.7856   s!=y 0.8285   gap -0.0430

Both interventions raise deployment accuracy from 67.3% to 80.7% (better overall, not just fairer, because the model is finally forced to learn the real signal), and both collapse the 62.5-point subgroup gap to a few points. Note the fine print, though: dropping s leaves a gap of -1.0 point, inside test-set noise (the model never sees s), while reweighting overshoots to -4.3 points, meaning the formerly disadvantaged group now does slightly better; the model still sees s, and the correction leans a little past neutral. Mitigation is measurable improvement, not absolution, and the measurement has to be the per-group table, every time.

Pitfall. In this synthetic world, deleting the s column works because s is the only trace of the attribute in the features. Real data is never so cooperative: sensitive attributes leak through proxies (zip code, names, school attended, word choice, purchasing history), and a model denied the attribute reconstructs it from the proxies to exploit the same shortcut. “We don’t collect that field” is a data-governance statement, not a fairness guarantee. The test that actually settles the question is behavioral, not architectural: evaluate per group (which requires knowing the attribute at evaluation time, even if the model never sees it) and check whether the gap survived.

One more honest caveat: our sandbox had a stable “true signal” to fall back on. When the historical labels themselves encode biased decisions (when y was produced by a process that already discriminated), no reweighting of features fixes it, because the target is the problem. That case is a data-collection and problem-formulation issue, and no downstream training trick converts a biased label into an unbiased one.

Models remember: membership inference

Bias is a harm a model does with its outputs; the next sections concern model memory. Membership inference and training-data extraction are related manifestations of memorization, but one does not mechanically imply the other for every model and threat model. Treat membership success as a warning signal that motivates a separate extraction audit.

The simplest attack, due to Yeom and colleagues (2018), needs no sophistication at all: compute the model’s loss on the candidate point, and guess “member” if the loss is low. It works exactly to the degree the model treats training data differently from fresh data — that is, exactly to the degree it overfits. That predicts a countermeasure for free: anything that closes the train-test gap should blunt the attack. The experiment tests the prediction on a small, noisily labeled dataset that a large MLP can memorize, comparing an overfit model, an early-stopped one, and a weight-decayed one (regularization reappearing as a privacy defense).

import torch
import torch.nn as nn

torch.manual_seed(0)

# Two-moons classification with label noise, so memorization has something to grab.
def moons(n, gen):
    t = torch.rand(n, generator=gen) * torch.pi
    x = torch.where(
        (torch.arange(n) % 2 == 0).unsqueeze(1),
        torch.stack([torch.cos(t), torch.sin(t)], dim=1),
        torch.stack([1 - torch.cos(t), 0.5 - torch.sin(t)], dim=1),
    ) + 0.2 * torch.randn(n, 2, generator=gen)
    y = (torch.arange(n) % 2).long()
    flip = torch.rand(n, generator=gen) < 0.1          # 10% mislabeled
    return x, torch.where(flip, 1 - y, y)

g = torch.Generator().manual_seed(0)
x_tr, y_tr = moons(150, g)      # small training set: easy to memorize
x_out, y_out = moons(150, g)    # non-members from the same distribution

def train(steps, weight_decay):
    torch.manual_seed(0)
    net = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(),
                        nn.Linear(256, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-3, weight_decay=weight_decay)
    for step in range(steps):
        loss = nn.functional.cross_entropy(net(x_tr), y_tr)
        opt.zero_grad(); loss.backward(); opt.step()
    return net

def attack(net):
    """Loss-threshold membership inference: guess 'member' when the model's
    loss on the example is below a threshold (best threshold reported)."""
    with torch.no_grad():
        l_in = nn.functional.cross_entropy(net(x_tr), y_tr, reduction="none")
        l_out = nn.functional.cross_entropy(net(x_out), y_out, reduction="none")
    accs = []
    for thresh in torch.cat([l_in, l_out]).sort().values:
        hits = (l_in <= thresh).float().sum() + (l_out > thresh).float().sum()
        accs.append(hits / (len(l_in) + len(l_out)))
    acc = torch.stack(accs).max()
    with torch.no_grad():
        tr_acc = (net(x_tr).argmax(1) == y_tr).float().mean()
        te_acc = (net(x_out).argmax(1) == y_out).float().mean()
    return acc.item(), tr_acc.item(), te_acc.item(), l_in.mean().item(), l_out.mean().item()

print(f"{'model':>22} {'attack acc':>11} {'train acc':>10} {'test acc':>9} "
      f"{'loss in':>8} {'loss out':>9}")
for name, steps, wd in [("overfit (5k steps)", 5000, 0.0),
                        ("early stop (300)", 300, 0.0),
                        ("weight decay 1e-2", 5000, 1e-2)]:
    net = train(steps, wd)
    a, tr, te, li, lo = attack(net)
    print(f"{name:>22} {a:>11.3f} {tr:>10.3f} {te:>9.3f} {li:>8.4f} {lo:>9.4f}")
print("(attack acc 0.5 = coin flip; the attacker knows nothing)")

Running this prints:

                 model  attack acc  train acc  test acc  loss in  loss out
    overfit (5k steps)       0.657      1.000     0.713   0.0169    3.3412
      early stop (300)       0.610      0.907     0.813   0.2178    0.6122
     weight decay 1e-2       0.540      0.840     0.807   0.3956    0.5001
(attack acc 0.5 = coin flip; the attacker knows nothing)

The mechanism could not be more visible. The memorizing model carries a 60-fold gap between its loss on members (0.017) and non-members (3.34), and the attacker converts that gap into 65.7% inference accuracy — far from omniscience, but a real leak: across a large population of candidates, 15 extra percentage points over coin-flipping identifies many individuals. Early stopping narrows the gap and the attack drops to 61%; weight decay narrows it further and the attack lands at 54%, close to chance; note the defended models are also simply better models (test accuracy 0.81 versus 0.71). On this toy, privacy and generalization point the same direction; the alignment is genuine but not a law, because at scale, models memorize rare examples even while generalizing well on average, which is exactly where the serious demonstrations live. Carlini and colleagues (USENIX Security 2021) ran the scaled version against GPT-2: from about 1,800 candidate generations selected by membership-style scoring, they verified over 600 as verbatim training-set content, including individuals’ names, contact information, and unique identifiers. Later work extended extraction to diffusion image generators. The toy and the production attack differ in machinery, not in kind.

Differential privacy: a knob with a price

Membership inference has a principled countermeasure with a rare property: a mathematical guarantee. Differential privacy (DP), formulated by Dwork, McSherry, Nissim, and Smith in 2006, defines privacy as insensitivity to any one person: a randomized computation is \(\varepsilon\)-differentially private if changing any single individual’s record changes the probability of any output by at most a factor of \(e^\varepsilon\). Small \(\varepsilon\) means no attacker (regardless of resources or side information) can learn much about any individual from the output, membership included. The guarantee is bought with noise, and the simplest purchase is the Laplace mechanism: to release a statistic, add Laplace noise scaled to sensitivity / epsilon, where sensitivity is the most one person could change the statistic. Which neighboring-dataset definition you use matters: the fraction statistic has sensitivity \(1/n\) under a fixed-size replace-one adjacency, while add/remove adjacency and a data-dependent denominator require different accounting. Always state adjacency, clipping/bounds, \(\varepsilon\), and (for approximate DP) \(\delta\) with the result. The script prices the trade for the fraction-with-attribute statistic:

import torch

torch.manual_seed(0)

# The statistic to release: what fraction of a cohort has some sensitive attribute.
# Changing ONE person changes the fraction by at most 1/n -- the "sensitivity".
# The Laplace mechanism adds noise scaled to sensitivity / epsilon.
def released_error(n, epsilon, trials=20000):
    sensitivity = 1.0 / n
    scale = sensitivity / epsilon
    noise = torch.distributions.Laplace(0.0, scale).sample((trials,))
    return noise.abs().median().item()   # typical |error| added to the true fraction

print("typical error added to a released fraction (median |noise|)")
print(f"{'epsilon':>8} {'n=100':>9} {'n=1,000':>9} {'n=100,000':>10}")
for eps in [0.01, 0.1, 1.0, 10.0]:
    row = [released_error(n, eps) for n in [100, 1000, 100000]]
    print(f"{eps:>8.2f} {row[0]:>9.4f} {row[1]:>9.4f} {row[2]:>10.6f}")

# What epsilon means: likelihood ratio for any output is bounded by e^epsilon.
for eps in [0.1, 1.0, 10.0]:
    print(f"epsilon = {eps:>5.1f}: an adversary's odds about any one person "
          f"change by at most a factor of {torch.tensor(eps).exp().item():.2f}")

Running this prints:

typical error added to a released fraction (median |noise|)
 epsilon     n=100   n=1,000  n=100,000
    0.01    0.6845    0.0691   0.000678
    0.10    0.0690    0.0069   0.000070
    1.00    0.0069    0.0007   0.000007
   10.00    0.0007    0.0001   0.000001
epsilon =   0.1: an adversary's odds about any one person change by at most a factor of 1.11
epsilon =   1.0: an adversary's odds about any one person change by at most a factor of 2.72
epsilon =  10.0: an adversary's odds about any one person change by at most a factor of 22026.46

Read the table along both axes. Down a column, privacy costs accuracy: at \(n = 100\), strong privacy (\(\varepsilon = 0.01\)) adds a typical error of 0.68 to a quantity that lives in \([0, 1]\) (the released number is garbage), while weak privacy (\(\varepsilon = 10\)) is nearly free. Across a row, population size is the great reconciler: at \(n = 100{,}000\), even \(\varepsilon = 0.01\) costs under a thousandth of a point. Privacy is cheap for statements about large groups and expensive for statements about small ones — which is the correct outcome, since statements about small groups are the dangerous ones. The last three lines translate \(\varepsilon\) into its actual meaning, and expose how much of the field’s practice rides on that translation: \(\varepsilon = 0.1\) bounds an adversary’s odds shift at 11%; \(\varepsilon = 10\) (a value that appears in real deployments) bounds it at a factor of 22,026, a “guarantee” doing little work beyond structure. When a system advertises DP, the honest follow-up question is always the value of \(\varepsilon\) and what it was computed over.

Two deployments anchor the concept in practice. The 2020 United States Census used a differential-privacy-based disclosure-avoidance system (the TopDown algorithm) to protect respondents in its published tables — a decision whose accuracy trade-offs for small communities were debated in court and in the statistics literature, exactly the debate the table above predicts. And for deep learning specifically, DP-SGD (Abadi et al., CCS 2016) makes the training process differentially private: clip each example’s gradient (bounding one person’s influence, the sensitivity) and add Gaussian noise to each batch’s sum, with an accounting method tracking the accumulated \(\varepsilon\) across steps. The clipping-plus-noise price is real accuracy loss at strong guarantees, which is why DP training remains standard practice mainly where the data is unambiguously sensitive. Note the pattern shared by the mitigations in this section: the fix has a dial (reweighting strength, regularization, \(\varepsilon\)), the dial has a cost, and the responsible move is to measure and report both ends, not to claim the dial’s existence as absolution.

Write it down: datasheets and model cards

The demonstrations above required knowing things about the data: which attribute was spuriously correlated, what the training distribution’s quirks were, whom the model would be applied to. In practice, that knowledge evaporates — datasets outlive their collectors, models get reused for tasks their builders never imagined, and the person deploying a model is rarely the person who trained it. The documentation movement of the late 2010s is best understood as an attack on this knowledge loss.

Two papers anchor it. Datasheets for Datasets (Gebru et al., 2018; published in CACM 2021) proposed that every dataset ship with a structured document (motivation, collection process, composition, known skews, recommended and discouraged uses) modeled on the datasheets that accompany electronic components. Model Cards for Model Reporting (Mitchell et al., FAT* 2019 (now FAccT)) did the same for trained models: intended use and out-of-scope uses, training data, evaluation results disaggregated by relevant groups (the paper’s central prescription, and exactly the per-group table our demo needed), and known limitations. Model cards are now standard practice on Hugging Face model repositories and in most serious model releases, which means you have almost certainly read one; the practical skill is writing one that says something — the discipline of stating what you measured, on whom, and what you did not measure. Your semester project report includes a model card for exactly this reason (template in the project guide; see the syllabus).

People. Timnit Gebru led the datasheets paper, and Gebru and Margaret Mitchell are authors on the model-cards work. Gebru also co-authored Gender Shades with Joy Buolamwini, reporting large intersectional disparities in commercial gender classification. The audit became an influential template and prompted public vendor responses; attributing a quantified causal improvement to the paper alone would require a separate longitudinal study.

Releasing: evaluation for failure, then a decision

Everything in the evaluation section measured average-case performance. Safety evaluation inverts the stance: instead of asking “how well does the model do on typical inputs?”, ask “how badly can it be made to do, by whom, at what cost?” The generic name for this is red-teaming — dedicating people (or, increasingly, other models) to eliciting failures: harmful outputs, private data regurgitation, jailbreaks of safety training, subgroup failures like the ones above. The essential cultural point is that a red team’s success metric is finding failures, which cures the confirmation bias built into ordinary benchmarking. For language models, the practice has been systematized: Perez et al. (2022) generate adversarial test cases with a language model, and Ganguli et al. (2022) document large-scale human red-teaming methodology. But the concept scales down to course-project size: attack your own model with distribution shift, adversarial inputs, and per-group splits before you claim it works.

A working taxonomy helps organize what a red team is actually looking for, because “failures” is not one category but at least five, each with its own methods. (1) Harmful output elicitation: prompting strategies that produce content the system is designed to refuse (the classic jailbreak, including role-play framings and multi-step decompositions). (2) Prompt injection: for systems that process untrusted text (documents, web pages, tool outputs), instructions smuggled into that text that hijack the model’s behavior, a category that grows in importance as models get connected to tools and email. (3) Privacy attacks: the membership-inference and extraction family you just ran, plus attempts to elicit memorized personal data by prompting. (4) Distribution shift and subgroup failure: the quieter failures of this section’s first half — performance cliffs on populations or conditions the evaluation missed. (5) Dangerous-capability evaluation: structured tests of whether a model can meaningfully assist with serious harms (cyber-offense, biological weapons uplift), which frontier labs now run before release under published internal policies. Categories 1–4 are directly exercisable at course scale; category 5 is included so you can read release documentation and know what the accompanying evaluations are for.

The practice has begun to standardize. In the United States, NIST’s AI Risk Management Framework (released January 2023) gives organizations a shared vocabulary for mapping, measuring, and managing AI risks, and its 2024 generative-AI companion profile translates the framework for exactly the systems these final readings cover. Regulation is arriving too: the European Union’s AI Act entered into force in 2024, with obligations for general-purpose model providers phasing in from August 2025 — including technical documentation and training-data summaries that look, not coincidentally, like formalized model cards and datasheets. The documentation habits this section teaches are becoming legal artifacts, not just professional courtesies.

A release decision is an engineering decision under documented uncertainty. Major frontier developers often publish some combination of model/system cards, safety evaluations, and staged or gated access, but coverage and transparency vary widely; “every serious lab” is not an auditable standard.

Historical note. The release-strategy debate has a clear origin point: OpenAI’s GPT-2 in February 2019. Citing misuse concerns (mass-produced disinformation chief among them), OpenAI initially published the paper with only a small model, then released progressively larger versions over nine months as evidence about misuse accumulated, with the full model arriving that November. The decision was widely criticized from both directions (too cautious; not cautious enough), and that argument, more than the model itself, is its legacy: it made “how should this be released?” a standard question that model builders are expected to answer explicitly.

Going deeper (Part B)

The documentation ecosystem grew up. Model cards, proposed in 2019 as a one-page discipline, have differentiated into an ecosystem. On the open side, model cards are the standard front page of every Hugging Face repository: structured, templated, and increasingly machine-readable. On the frontier side, releases now ship system cards: book-length documents (GPT-4’s 2023 system card is an early landmark, and major labs have published equivalents since) covering red-teaming results, dangerous-capability evaluations, mitigations, and known failure modes for the deployed system (model plus safety training plus filters plus policy) rather than the bare network. The shift from “model” to “system” in the name is substantive: most deployed behavior, good and bad, is a property of the whole stack. Alongside cards, incident reporting has emerged on the model of aviation safety: the AI Incident Database, launched under the Partnership on AI in 2020, collects publicly reported harms from deployed systems so that builders can learn from failures they did not personally cause. Browsing it is a fast education in how models actually fail in the world — rarely in the ways their builders’ benchmarks measured.

“Open source” became a contested term. Through the 2010s, “open-sourcing a model” loosely meant publishing weights. The 2020s made the looseness untenable, because the artifacts diverged: a release may include weights but not training data or code, and the license may forbid uses or users. Llama-family models, for instance, ship weights under a community license with use restrictions (including a clause requiring companies above a user-count threshold to obtain a separate license), and the Open Source Initiative has stated publicly that such licenses do not meet open-source standards. The OSI drew the line formally in October 2024 with the Open Source AI Definition 1.0, requiring usable code, weights under open terms, and enough training-data information to substantially recreate the system — a bar few flagship “open” models clear, which was the point of writing it down. In between sit the RAIL-family licenses (used by early Stable Diffusion releases, among others), which are permissive except for enumerated prohibited uses, trading classical openness for a behavioral rider of debatable enforceability. And some models (several Mistral releases, for example) do ship under plain Apache 2.0. For your own work the practical rule is unglamorous: the license of a model (and of its training data) is part of its technical specification. Read it before you build on it, and state your own project’s terms in its model card.

Where the open questions actually live. It would misrepresent the field to end with settled procedure, because the deepest issues here are unsettled by design. Per-group evaluation requires defining groups, and fairness metrics conflict with one another mathematically: work in the late 2010s proved that natural criteria (calibration within groups, equal false-positive and false-negative rates) cannot generally hold simultaneously, so “the fair model” is not a well-posed engineering target; choosing which criterion matters is a decision about values that the model card should record rather than hide. Release norms remain in genuine flux: the field runs a live experiment between open-weights ecosystems (auditable by anyone, controllable by no one) and gated APIs (controllable, but concentrating power and limiting scrutiny), and serious people disagree about which failure mode is worse. What has stabilized is the process expectation this section teaches: measure disaggregated, attack before release, document what you found and what you did not measure, and treat the release itself as a decision with alternatives. Those habits transfer to whatever equilibrium the field reaches — and they are the part you are graded on.

Common pitfalls (Part B)

  • Aggregate accuracy is acceptable, so the model ships — then specific user groups report systematic failures. Cause: aggregate metrics average away concentrated harm; our demo’s 67% hid a worse-than-chance subgroup. Fix: disaggregate every headline metric by the groups the model card claims to serve, and gate release on the worst group, not the mean.
  • The bias evaluation passes because the test set has the same skew as the training set. Cause: an i.i.d. test set inherits the spurious correlation, so the shortcut scores well on it. Fix: evaluate under the deployment distribution, or construct balanced/shifted test splits where the correlation is broken — that is what exposed our model.
  • The sensitive attribute was removed, so the team declares the model fair. Cause: proxies reconstruct the attribute; behavior, not architecture, is what needs testing. Fix: run per-group evaluations anyway, and probe whether the attribute is predictable from the model’s inputs or representations.
  • The model card is written the week of release, from memory. Cause: documentation treated as marketing rather than engineering record. Fix: start the card when training starts; record data sources, skews, and evaluation gaps as they are discovered — they are cheapest to capture in the moment.
  • A model trained on sensitive records ships because “the data never leaves the server.” Cause: conflating data access control with model privacy — the model itself leaks membership and, at scale, verbatim content, as the attack above demonstrates. Fix: measure the train-test loss gap and attack accuracy as part of evaluation; where the data warrants it, train with DP-SGD and report the actual \(\varepsilon\).
  • A project builds on a model because its page said “open,” and licensing problems surface at release time. Cause: “open weights” spans everything from Apache 2.0 to use-restricted community licenses. Fix: read the license as part of the technical evaluation, check the restrictions against your use, and record the terms (and your own) in the model card.

Try it yourself (Part B)

  1. Add a proxy feature to the sandbox: proxy = s with 10% of values flipped, then drop s but keep the proxy. How much of the subgroup gap returns? Measure the per-group table for baseline, drop-s, and drop-both.
  2. Sweep the training correlation p_match over 0.6, 0.7, 0.8, 0.9, 0.95 and plot (or tabulate) the learned weight on s and the deployment-set subgroup gap. At what correlation strength does the shortcut start dominating the real signal?
  3. Implement a simple worst-group training loop: after each epoch, upweight the (y, s) cell with the lowest training accuracy, and compare its final per-group table to the static reweighting above.
  4. Write a one-page model card for the baseline shortcut classifier as if it were about to be deployed for screening: intended use, training data description (including the 0.9 correlation), disaggregated evaluation, and out-of-scope uses. Then revise it for the reweighted model. Which sections changed?
  5. In the membership-inference experiment, sweep the training-set size over 150, 500, and 2000 (keeping everything else fixed) and tabulate attack accuracy against the train-test gap. Does more data protect privacy here, and through what mechanism?
  6. Simulate a crude DP defense for the membership attack: add Gaussian noise of increasing scale to every logit before computing the attack’s losses, and plot attack accuracy versus the model’s test accuracy as noise grows. Where is the knee, and what does this teach about the epsilon dial?

Further reading (Part B)

  • Mitchell et al., “Model Cards for Model Reporting,” FAT* 2019 (now FAccT) — the disaggregated-evaluation prescription this section’s demos motivate; your project’s model card template descends from it. arxiv.org/abs/1810.03993
  • Gebru et al., “Datasheets for Datasets,” CACM 2021 — structured documentation for datasets, modeled on electronic-component datasheets. arxiv.org/abs/1803.09010
  • Buolamwini & Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,” FAccT 2018 — the per-group audit that moved an industry. proceedings.mlr.press/v81/buolamwini18a.html
  • Geirhos et al., “Shortcut Learning in Deep Neural Networks,” Nature Machine Intelligence 2020 — the survey placing our two-feature demo in a much larger family. arxiv.org/abs/2004.07780
  • Carlini et al., “Extracting Training Data from Large Language Models,” USENIX Security 2021 — verbatim training-data extraction from GPT-2; the scaled version of our membership attack. arxiv.org/abs/2012.07805
  • Abadi et al., “Deep Learning with Differential Privacy” (DP-SGD), CCS 2016 — gradient clipping plus noise, with the privacy accounting that makes \(\varepsilon\) computable for training. arxiv.org/abs/1607.00133
  • Ganguli et al., “Red Teaming Language Models to Reduce Harms,” 2022 — large-scale human red-teaming methodology, with data. arxiv.org/abs/2209.07858
  • NIST AI Risk Management Framework — the shared risk vocabulary referenced above, with its generative-AI profile. nist.gov/itl/ai-risk-management-framework
  • Open Source Initiative, “The Open Source AI Definition 1.0,” 2024 — where the “open source model” line is now drawn, and why most open-weight releases fall short of it. opensource.org/ai
  • Hugging Face documentation on model cards — a practical template to write yours against. huggingface.co/docs/hub/model-cards

Before class you should be able to

  • Name the subtle forms of leakage (preprocessing before the split, duplicates, temporal leakage, benchmark contamination) and the symptom they share.
  • Explain why a single-run comparison of two configs is worthless, and what “mean +/- std over seeds” adds.
  • Describe the bootstrap procedure for a confidence interval on test accuracy, and read a CI against a claimed improvement.
  • Explain the multiple-comparisons trap in a hyperparameter sweep and state one defense (report the whole sweep; confirm the winner on fresh seeds; Bonferroni).
  • Given the shortcut-learning setup, predict what the classifier does with a spuriously correlated attribute and why aggregate accuracy hides the harm.
  • Explain the loss-threshold membership-inference attack and why regularization blunts it.
  • State what a model card contains and why evaluation must be disaggregated by group.
  • Say what red-teaming inverts relative to ordinary benchmarking.

Connects to

  • Quiz 4 takes the final 30 minutes of this class (6:45–7:15 PM, in Canvas): the last module quiz, covering Module 4 (S23–S26).
  • HW4 is due at S29, the last class meeting: its evaluation half (bootstrap CIs and a paired model comparison, NumPy only) is section 4.5 turned into an assignment.
  • Your project report requires a model card and is due at the end of this week, before the recorded presentation; the syllabus’s fair-comparison policy is this reading turned into a grading rubric; a code defense (at S28 or S29, for the students the instructor calls) may include questions about your evaluation choices.

← S26 · Efficiency and Scale · All readings