Module 3 · Reading 3.5 · read before S18: LLM Pretraining

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

A Transformer fresh out of the constructor has random weights and uninformative predictions. Pretraining is a large optimization process over next-token or related self-supervised objectives. Corpora may be sampled with repetition, mixtures are often reweighted, and high-quality sources may be deliberately oversampled; it is not necessarily one literal pass over each document. This section explains the objective, sketches where the data comes from, and then turns to the question that has shaped the field’s last five years: given a fixed compute budget, how big should the model be and how much data should it see? Kaplan et al. (2020) and Hoffmann et al. (2022, “Chinchilla”) answered it differently, and the difference changed how subsequent models were trained.

You will also run a miniature version of a scaling experiment yourself, and we will look critically at the claim that scale produces “emergent” capabilities.

How to read this

One objective: predict the next token

The pretraining objective for GPT-style models is exactly the language-modeling loss you met when training tiny character models: given tokens \(x_{1..t-1}\), output a probability distribution over the next token, and minimize the average negative log-likelihood

\[L = -\frac{1}{T} \sum_t \log p(x_t \mid x_{1..t-1})\]

That is it. There is no labeled dataset, no human annotation, no task-specific head. The labels are the text itself, shifted by one position — which is why this is called self-supervised learning. Every position in every document contributes a training signal, so a corpus of a trillion tokens provides a trillion prediction problems for free.

Why is such a plain objective enough? The informal argument: predicting the next token well forces the model to absorb whatever regularities help prediction. Predicting the next word of "the capital of Idaho is" requires a fact. Predicting the next line of a Python function requires understanding the code above it. Predicting the closing argument of an essay requires tracking the argument. Compression and prediction are two faces of the same coin, and the cross-entropy loss literally measures how well the model compresses text. None of this guarantees the model reasons the way people do — it guarantees only that the model’s predictions match the statistics of its training distribution, a distinction that matters later when we discuss evaluation.

Mechanically, pretraining is the same training loop you have written since the S08 reading: sample a batch of token sequences, run the Transformer with a causal mask so position \(t\) cannot see position \(t+1\), compute cross-entropy at every position, backpropagate, step AdamW. What changes at scale is everything around the loop (thousands of accelerators, model and data parallelism, checkpointing), but not the loop itself.

Historical note. Pretraining-then-adapting was not always the default. Through the mid-2010s, NLP models were mostly trained per task from scratch, with pretrained word vectors (word2vec, 2013; GloVe, 2014) as the only shared component. The 2018 generation (ELMo, GPT, and BERT) showed that pretraining an entire network on unlabeled text and then adapting it beat task-specific architectures across the board, and the 2020 GPT-3 paper pushed further: with enough scale, many tasks could be performed with no gradient updates at all. Within about three years, “train a model for your task” became “adapt a pretrained model to your task” almost everywhere in NLP.

Where the data comes from

Trillion-token corpora are not curated by hand. A typical pipeline, described at varying levels of detail in papers such as GPT-3 (Brown et al., 2020), T5’s C4 corpus (Raffel et al., 2020), and The Pile (Gao et al., 2020), looks like this:

  1. Crawl. Start from a web crawl (often Common Crawl, a public archive of billions of web pages), plus higher-quality sources: books, encyclopedias, code repositories, scientific papers.
  2. Extract and clean. Strip HTML boilerplate, navigation menus, and ads; detect language; remove documents that are too short, too repetitive, or mostly symbols.
  3. Filter for quality. Heuristics (punctuation ratios, word lists) or a classifier trained to distinguish “reference-quality” text from random crawl text. This step is where a great deal of judgment (and potential bias) enters.
  4. Deduplicate. Remove exact and near-duplicate documents. Duplicated text wastes compute and, worse, encourages memorization; deduplication also reduces the chance that benchmark test sets leak into training data.
  5. Mix. Decide how many epochs (often less than one) each source gets. High-quality sources are commonly up-weighted relative to raw crawl data.

Two practical points deserve emphasis. First, the tokenizer from the previous section is fixed before pretraining and frozen forever after, so data decisions and tokenizer decisions interact: a corpus with little code produces a tokenizer that splits code inefficiently. Second, data quality work is unglamorous but repeatedly shown to matter as much as architecture; much of it is documented only partially in public papers, which is worth remembering when comparing models.

The mixing step deserves its own paragraph, because it hides real decisions. A “mixture” assigns each source a sampling weight, and the weights are not proportional to size: the GPT-3 paper reports that its filtered Common Crawl made up 60% of the training mixture while Wikipedia was sampled for multiple effective epochs. Upweighting is a quality bet with overfitting risk. When high-quality data runs short, Muennighoff et al. report that, in their tested fixed-compute regimes, repetition through several epochs incurred relatively little penalty before diminishing returns became substantial. The result is conditional on their data, scale, and budget — not a universal free-epoch rule.

Deduplication you can run

Step 4 of the pipeline, deduplication, is worth slowing down for, because it is both consequential and very implementable. Lee et al. (2022, “Deduplicating Training Data Makes Language Models Better”) audited standard corpora and report finding, among other things, a single 61-word sentence repeated over 60,000 times in C4; they also report that models trained on deduplicated data emit memorized training text about ten times less often, at equal or better validation quality. Duplication has a second, sneakier cost: if a benchmark’s test questions are duplicated into the training crawl, evaluation silently becomes memorization — contamination, a problem we return to in evaluation.

Exact duplicates are easy to hash after a chosen normalization. Near-duplicates motivate MinHash, because comparing all pairs of documents is quadratic in corpus size. Represent each document as the set of its character or word n-grams (shingles); the overlap between two documents is their Jaccard similarity, \(J(A, B) = \lvert A \cap B\rvert / \lvert A \cup B\rvert\). Under an ideal uniformly random permutation of the shingle universe, the collision probability of the minimum is exactly the Jaccard similarity. Practical implementations use hash functions as efficient approximations to independent minwise permutations, so the estimator is approximate for both finite \(k\) and non-ideal hashes. Locality-sensitive banding avoids an all-pairs scan.

# Exact and near-duplicate detection on a toy corpus.
# Exact dedup: hash a normalized copy of each document.
# Near-dup detection: MinHash — estimate Jaccard similarity of shingle sets
# from k independent minimum hash values, without comparing sets directly.
import hashlib
import random

random.seed(0)

docs = [
    "The quick brown fox jumps over the lazy dog.",
    "Deep learning models improve with more data and compute.",
    "The quick brown fox jumps over the lazy dog.",          # exact dup of 0
    "  the QUICK brown fox jumps over the lazy dog. ",       # dup after normalizing
    "Deep learning models improve with more data & compute.",  # near-dup of 1
    "Neural networks are trained with gradient descent.",
    "A completely unrelated sentence about lasagna recipes.",
]

def normalize(text):
    return " ".join(text.lower().split())

# --- Stage 1: exact deduplication via content hashing ----------------------
seen, kept = {}, []
for i, doc in enumerate(docs):
    h = hashlib.sha1(normalize(doc).encode()).hexdigest()
    if h in seen:
        print(f"doc {i} is an exact duplicate of doc {seen[h]} (after normalizing)")
    else:
        seen[h] = i
        kept.append(i)
print(f"exact dedup: {len(docs)} docs -> {len(kept)} docs {kept}")

# --- Stage 2: MinHash near-duplicate estimation on the survivors -----------
def shingles(text, n=3):
    """Character n-grams of the normalized text, as a set."""
    t = normalize(text)
    return {t[i:i + n] for i in range(len(t) - n + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

K = 128  # number of hash functions = signature length
MERSENNE = (1 << 61) - 1
hash_params = [(random.randrange(1, MERSENNE), random.randrange(MERSENNE))
               for _ in range(K)]

def minhash_signature(shingle_set):
    """For each of K hash functions, keep the minimum hash over the set."""
    ints = [int.from_bytes(hashlib.sha1(s.encode()).digest()[:8], "big")
            for s in shingle_set]
    return [min((a * x + b) % MERSENNE for x in ints) for a, b in hash_params]

def minhash_estimate(sig_a, sig_b):
    """P[min-hash matches] = Jaccard similarity, so the match rate estimates it."""
    return sum(x == y for x, y in zip(sig_a, sig_b)) / K

sigs = {i: minhash_signature(shingles(docs[i])) for i in kept}
print(f"\npairwise similarity among the {len(kept)} surviving docs (threshold 0.6):")
print(f"{'pair':>8} {'true Jaccard':>13} {'MinHash est.':>13}  verdict")
for ai in range(len(kept)):
    for bi in range(ai + 1, len(kept)):
        i, j = kept[ai], kept[bi]
        true_j = jaccard(shingles(docs[i]), shingles(docs[j]))
        est_j = minhash_estimate(sigs[i], sigs[j])
        verdict = "NEAR-DUP" if est_j >= 0.6 else "-"
        if true_j > 0.05 or est_j >= 0.6:
            print(f"  ({i}, {j}) {true_j:>13.4f} {est_j:>13.4f}  {verdict}")

Running this prints:

doc 2 is an exact duplicate of doc 0 (after normalizing)
doc 3 is an exact duplicate of doc 0 (after normalizing)
exact dedup: 7 docs -> 5 docs [0, 1, 4, 5, 6]

pairwise similarity among the 5 surviving docs (threshold 0.6):
    pair  true Jaccard  MinHash est.  verdict
  (1, 4)        0.8571        0.8516  NEAR-DUP
  (1, 5)        0.0526        0.0391  -
  (4, 5)        0.0538        0.0469  -

Trace the two stages. Exact hashing catches byte-identical and normalization-equivalent documents; MinHash estimates near-duplicate overlap. Production choices vary: signatures commonly use dozens to hundreds of hash components, with more components buying lower estimator variance at greater storage and compute. Thresholds, shingle definitions, normalization, and banding are all policy choices.

Pitfall. Deduplication thresholds are silent policy decisions. Set the near-duplicate threshold too low and you delete legitimate reuse (boilerplate licenses, famous speeches, code idioms), distorting the distribution; too high and templated spam floods the corpus. And no within-corpus dedup protects you from cross-corpus leakage: the benchmark you plan to evaluate on may already be inside the crawl. Before trusting any benchmark number, ask what the training corpus was and what decontamination the authors report — for many closed models, the honest answer is that you cannot check.

Scaling laws: Kaplan (2020)

By 2019 it was clear that bigger models trained on more data kept getting better, but “bigger is better” is not an engineering guideline. Kaplan et al. (2020, “Scaling Laws for Neural Language Models”) made the observation quantitative. Training many Transformer language models of different sizes, they reported that test loss falls as a smooth power law in each of three quantities, over several orders of magnitude: number of non-embedding parameters \(N\), dataset size in tokens \(D\), and training compute \(C\). A power law means straight lines on a log-log plot:

\[L(N) \sim \left(\frac{N_c}{N}\right)^{\alpha_N}\]

with small exponents: the paper fits roughly \(\alpha_N \approx 0.076\) for parameters and \(\alpha_D \approx 0.095\) for data. Small exponents mean progress is real but expensive: cutting the loss meaningfully requires multiplying resources, not adding to them.

The paper’s most consequential claim was about allocation. Given a fixed compute budget \(C\), how should you split it between a bigger model and more tokens? Using the approximation \(C \approx 6ND\) floating-point operations (about 6 FLOPs per parameter per token for a forward-plus-backward pass), Kaplan et al. reported that compute-optimal training grows the model much faster than the data (roughly \(N \propto C^{0.73}\) versus \(D \propto C^{0.27}\)), and that very large models should be stopped well short of convergence. The field listened: GPT-3 (2020) had 175 billion parameters and was trained on roughly 300 billion tokens, less than two tokens per parameter.

Because the methodology is exactly what the later dispute turns on, note three specifics of the Kaplan et al. setup. First, their \(N\) counts non-embedding parameters only — embeddings are excluded from both the parameter count and the compute estimate, a choice that matters most for the small models that anchor the low end of the fit. Second, their runs shared training hyperparameters across scales rather than re-tuning per scale; in particular, learning-rate schedule and warmup were not adjusted to each run’s token budget. Third, the compute-optimal frontier was estimated largely from early stopping points along training curves: train models of various sizes, and for each compute level ask which size has the lowest loss at that moment. Each choice is defensible; together, the later literature argues, they tilt the fit toward “bigger model, fewer tokens.”

Compute accounting: where 6ND comes from

The estimate \(C \approx 6ND\) is used so often that it is worth deriving rather than memorizing. Multiplying a vector by an \(m \times n\) weight matrix costs about \(2mn\) FLOPs (one multiply and one add per weight). Summing that over every weight matrix in the network, one token’s forward pass costs about \(2N\) FLOPs: two per parameter. The backward pass computes two matmuls for every forward one (gradients with respect to inputs and to weights), costing roughly \(4N\). Total: about \(6N\) FLOPs per token, times \(D\) tokens. What this accounting ignores is everything that is not a weight multiplication: attention-score computation (which grows with sequence length and has no weights), normalization, softmax, activation functions. The script below does the layer-by-layer count for a decoder-only Transformer and shows when the ignored terms start to matter, then converts published budgets into wall-clock time on real hardware.

# Where does C ~= 6*N*D come from? Count the matrix-multiply FLOPs of one
# forward pass per token, layer by layer, and compare against 2*N (forward)
# and 6*N (forward + backward, since backward costs ~2x forward).
# A matmul that multiplies by an (m x n) weight costs ~2*m*n FLOPs per token.

def transformer_accounting(d, n_layer, vocab, seq):
    per_layer = {
        "attn QKV projections (3 * 2*d*d)": 6 * d * d,
        "attn output projection (2*d*d)":   2 * d * d,
        "attn scores QK^T (2*seq*d)":       2 * seq * d,
        "attn weighted sum AV (2*seq*d)":   2 * seq * d,
        "MLP up + down (2*d*4d + 2*4d*d)":  16 * d * d,
    }
    fwd = n_layer * sum(per_layer.values()) + 2 * d * vocab  # + LM head
    n_params = n_layer * 12 * d * d + vocab * d              # weights, tied embed
    return per_layer, fwd, n_params

configs = [("GPT-2 small-ish", 768, 12, 50257, 1024),
           ("7B-class",        4096, 32, 32000, 4096),
           ("7B-class, long ctx", 4096, 32, 32000, 131072)]

for name, d, n_layer, vocab, seq in configs:
    per_layer, fwd, n = transformer_accounting(d, n_layer, vocab, seq)
    print(f"{name}: d={d}, layers={n_layer}, seq={seq}")
    print(f"  params N ~ {n:.3e}   forward/token {fwd:.3e} FLOPs "
          f"(= {fwd / (2 * n):.2f} * 2N)")
    print(f"  fwd+bwd/token {3 * fwd:.3e} FLOPs (= {3 * fwd / (6 * n):.2f} * 6N)")

# The attention-score terms are the only ones that grow with seq, which is
# why 6*N*D is accurate at ordinary context lengths and drifts at long ones.

# --- From FLOPs to wall-clock: how long do real budgets take? --------------
PEAK = 312e12   # A100 bf16 peak, FLOP/s (NVIDIA datasheet)
MFU = 0.4       # optimistic sustained utilization for large-scale training

print(f"\n{'model':>22} {'N':>9} {'D':>9} {'tok/param':>10} {'C=6ND':>10} {'A100-days':>10}")
for name, N, D in [("Chinchilla (paper)", 70e9, 1.4e12),
                   ("LLaMA-7B (paper)", 7e9, 1.0e12),
                   ("Llama 2-7B (paper)", 7e9, 2.0e12)]:
    C = 6 * N * D
    days = C / (PEAK * MFU) / 86400
    print(f"{name:>22} {N:>9.1e} {D:>9.1e} {D / N:>10.1f} {C:>10.2e} {days:>10,.0f}")

Running this prints:

GPT-2 small-ish: d=768, layers=12, seq=1024
  params N ~ 1.235e+08   forward/token 2.848e+08 FLOPs (= 1.15 * 2N)
  fwd+bwd/token 8.544e+08 FLOPs (= 1.15 * 6N)
7B-class: d=4096, layers=32, seq=4096
  params N ~ 6.574e+09   forward/token 1.529e+10 FLOPs (= 1.16 * 2N)
  fwd+bwd/token 4.588e+10 FLOPs (= 1.16 * 6N)
7B-class, long ctx: d=4096, layers=32, seq=131072
  params N ~ 6.574e+09   forward/token 8.187e+10 FLOPs (= 6.23 * 2N)
  fwd+bwd/token 2.456e+11 FLOPs (= 6.23 * 6N)

                 model         N         D  tok/param      C=6ND  A100-days
    Chinchilla (paper)   7.0e+10   1.4e+12       20.0   5.88e+23     54,532
      LLaMA-7B (paper)   7.0e+09   1.0e+12      142.9   4.20e+22      3,895
    Llama 2-7B (paper)   7.0e+09   2.0e+12      285.7   8.40e+22      7,790

The first half validates the rule and shows its cracks. At ordinary context lengths (1k-4k), the full matmul count lands within about 16% of \(6N\); at 131k tokens, attention terms make the naive estimate fail badly. The second half uses an illustrative assumed 40% sustained utilization to convert Chinchilla’s 5.88e23 FLOPs into about 54,500 A100-days. Real MFU (model FLOPs utilization: the fraction of theoretical peak spent on the model’s required FLOPs) varies substantially with model shape, hardware, parallelism, sequence length, and measurement convention; replace 40% with a measured value for an actual plan.

Chinchilla (2022): the correction

Hoffmann et al. (2022, “Training Compute-Optimal Large Language Models”) re-ran the analysis with a different experimental design. The headline methodological change: for every run, the learning-rate schedule was set to decay over that run’s actual token budget, rather than reusing one fixed schedule regardless of training length. This matters because a model whose schedule is planned for 300 billion tokens but stopped at 30 billion is evaluated mid-decay, at a point its optimizer never intended — systematically overstating the loss of “small budget” configurations and thereby distorting where the optimum appears to lie. Hoffmann et al. also counted total parameters and total FLOPs rather than excluding embeddings, and trained over 400 models up to 16B parameters and 500B tokens.

Their analysis triangulates with three estimation approaches, and it is worth knowing them because you will meet them in later papers. Approach 1 fixes a set of model sizes, varies the number of training tokens for each, and reads the compute-optimal frontier off the envelope of training curves. Approach 2 constructs isoFLOP profiles: fix a total compute budget \(C\), train several model sizes to exactly that budget, and observe that loss versus model size traces a U-shape (a parabola in \(\log N\)) whose minimum is the optimal size for that budget; repeat across budgets and fit how the minima move. Approach 3 fits a parametric loss surface

\[L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}\]

(an irreducible-entropy term plus a capacity term plus a data term) to all runs jointly, and derives the optimal allocation analytically. The paper reports that all three approaches agree: compute-optimal \(N\) and \(D\) scale equally, both roughly as \(C^{0.5}\). In other words, every time you 10x your compute, you should make the model about 3x bigger and train it on about 3x more tokens — not 5x bigger on barely more data.

The paper’s headline demonstration: using the same training compute budget as the 280-billion-parameter Gopher model, they trained Chinchilla, a 70-billion-parameter model on 1.4 trillion tokens (4x smaller, 4x more data), and reported that it uniformly outperformed Gopher across their evaluations. From their fitted estimates comes the widely quoted rule of thumb of roughly 20 training tokens per parameter for compute-optimal training. Note what the rule optimizes: loss for a fixed training budget. If a model will be served to millions of users, inference cost dominates, and it can pay to train a smaller model on far more than 20 tokens per parameter — which is exactly what many post-2022 models (e.g., the LLaMA family, by its authors’ own account) chose to do.

The Kaplan-vs-Chinchilla episode is a good scientific parable: both papers fit power laws to real measurements, and the difference came down to experimental design details (learning-rate schedules, how “compute-optimal” points were selected). The postmortem eventually got quite precise: Porian et al. (2024, “Resolving Discrepancies in Compute-Optimal Scaling of Language Models”) reproduced the Kaplan-style analysis on open datasets and report that three factors account for the gap: the computational cost of the final (vocabulary-projection) layer, which Kaplan’s non-embedding accounting omits and which distorts the small-model end of the fit; warmup schedules that are too long for small runs, penalizing them; and the failure to re-tune optimizer hyperparameters as scale changes. Correcting the three, they report, recovers the Chinchilla exponents — and, interestingly, that careful learning-rate decay matching, the explanation most people repeated after 2022, was not essential. The moral stands either way: treat any scaling law as an empirical fit with a domain of validity, not a law of nature, and read the methods section before the exponents.

People. Jared Kaplan, a theoretical physicist at Johns Hopkins, co-authored the 2020 scaling-laws paper while consulting for OpenAI and brought a physicist’s habit of looking for power laws in messy data; he later co-founded Anthropic. The 2022 compute-optimal paper came from DeepMind, with Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and Laurent Sifre among its lead authors; Mensch went on to co-found Mistral AI, whose models followed the small-model-many-tokens philosophy. That the two camps reached different exponents from similar data illustrates how much experimental design shapes empirical “laws”.

A toy scaling experiment you can run

Real scaling-law papers train hundreds of models across many orders of magnitude of compute. We cannot do that in 30 seconds on a CPU, but we can observe the qualitative core: at a fixed data and step budget, bigger models reach lower loss. The script below generates text in a made-up language (50 pseudo-words built from 10 syllables), then trains the same one-block character-level Transformer at three widths and evaluates held-out loss.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# --- Generate a corpus in a made-up language (no downloads) ---------------
g = torch.Generator().manual_seed(0)
syllables = ["ba", "ku", "ri", "ta", "no", "shi", "mo", "ler", "vin", "da"]
words = []
for i in range(50):
    n_syl = int(torch.randint(2, 4, (1,), generator=g))
    idx = torch.randint(0, len(syllables), (n_syl,), generator=g)
    words.append("".join(syllables[j] for j in idx))
corpus = " ".join(words[int(torch.randint(0, 50, (1,), generator=g))]
                  for _ in range(20000))

chars = sorted(set(corpus))
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in corpus])
n_train = int(0.9 * len(data))
train_data, val_data = data[:n_train], data[n_train:]
print(f"corpus: {len(corpus)} chars, vocab size {len(chars)}")

CTX = 32

def get_batch(split_data, batch_size, gen):
    ix = torch.randint(0, len(split_data) - CTX - 1, (batch_size,), generator=gen)
    x = torch.stack([split_data[i:i + CTX] for i in ix])
    y = torch.stack([split_data[i + 1:i + CTX + 1] for i in ix])
    return x, y

# --- A minimal decoder-only Transformer -----------------------------------
class Block(nn.Module):
    def __init__(self, d, n_head):
        super().__init__()
        self.ln1 = nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, n_head, batch_first=True)
        self.ln2 = nn.LayerNorm(d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x):
        mask = nn.Transformer.generate_square_subsequent_mask(x.size(1))
        h = self.ln1(x)
        a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False)
        x = x + a
        return x + self.mlp(self.ln2(x))

class CharLM(nn.Module):
    def __init__(self, vocab, d, n_head, n_layer):
        super().__init__()
        self.tok = nn.Embedding(vocab, d)
        self.pos = nn.Embedding(CTX, d)
        self.blocks = nn.Sequential(*[Block(d, n_head) for _ in range(n_layer)])
        self.ln_f = nn.LayerNorm(d)
        self.head = nn.Linear(d, vocab)

    def forward(self, idx):
        x = self.tok(idx) + self.pos(torch.arange(idx.size(1)))
        return self.head(self.ln_f(self.blocks(x)))

def train_one(d, n_head, steps=300, batch_size=64):
    torch.manual_seed(0)
    gen = torch.Generator().manual_seed(1)
    model = CharLM(len(chars), d, n_head, n_layer=1)
    n_params = sum(p.numel() for p in model.parameters())
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
    for step in range(steps):
        x, y = get_batch(train_data, batch_size, gen)
        loss = F.cross_entropy(model(x).view(-1, len(chars)), y.view(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    with torch.no_grad():
        eval_gen = torch.Generator().manual_seed(2)
        losses = []
        for _ in range(20):
            x, y = get_batch(val_data, batch_size, eval_gen)
            losses.append(F.cross_entropy(
                model(x).view(-1, len(chars)), y.view(-1)).item())
    return n_params, sum(losses) / len(losses)

print(f"{'d_model':>8} {'params':>10} {'val loss (nats/char)':>22}")
for d, n_head in [(8, 2), (32, 4), (128, 4)]:
    n_params, val_loss = train_one(d, n_head)
    print(f"{d:>8} {n_params:>10,} {val_loss:>22.4f}")
print(f"uniform-guess baseline: {math.log(len(chars)):.4f} nats/char")

Running this prints:

corpus: 127760 chars, vocab size 17
 d_model     params   val loss (nats/char)
       8      1,433                 1.1419
      32     14,897                 0.7343
     128    206,993                 0.6366
uniform-guess baseline: 2.8332 nats/char

Held-out loss falls monotonically as the parameter count grows from 1.4 thousand to 207 thousand (a 144x size range), with everything else held fixed. Also note the diminishing returns: the first size jump buys 0.41 nats, the second (a bigger multiplicative jump) buys only 0.10. Be precise about what this toy run shows and does not show: it demonstrates that capacity limits loss on structured data, and it is consistent with smooth improvement under scaling. It is not a scaling-law measurement — three sizes, one seed, 300 steps, and a synthetic language cannot establish a power law, and we did not tune hyperparameters per size (which both Kaplan and Hoffmann show can change conclusions).

What a compute budget buys

The second script turns the published exponents into concrete sizing decisions. It anchors the power laws at Chinchilla’s own operating point (70e9 parameters, 1.4e12 tokens) and uses \(C \approx 6ND\).

# Compute-optimal sizing, anchored to the Chinchilla paper's own choice.
# Hoffmann et al. (2022) report N_opt ~ C^0.50 and D_opt ~ C^0.50, and trained
# Chinchilla with N = 70e9 parameters on D = 1.4e12 tokens. Using the standard
# approximation C ~= 6*N*D FLOPs, we anchor the power laws at that point.

N_CHINCHILLA = 70e9      # parameters
D_CHINCHILLA = 1.4e12    # training tokens
C_CHINCHILLA = 6 * N_CHINCHILLA * D_CHINCHILLA

def compute_optimal(C, a=0.50, b=0.50):
    N = N_CHINCHILLA * (C / C_CHINCHILLA) ** a
    D = D_CHINCHILLA * (C / C_CHINCHILLA) ** b
    return N, D

print(f"anchor: N = 70.0e9, D = 1.40e12  =>  C = 6*N*D = {C_CHINCHILLA:.2e} FLOPs")
print(f"{'C (FLOPs)':>12} {'N_opt (params)':>16} {'D_opt (tokens)':>16} {'tokens/param':>13}")
for exp in range(19, 26):
    C = 10.0 ** exp
    N, D = compute_optimal(C)
    print(f"{C:>12.0e} {N:>16.3e} {D:>16.3e} {D / N:>13.1f}")

# What did Kaplan et al. (2020) recommend instead? Their fits favored growing
# N much faster than D (roughly N ~ C^0.73, D ~ C^0.27). Anchoring those
# exponents at the same point shows how differently the two papers allocate
# a 100x larger budget:
for a, b, label in [(0.73, 0.27, "Kaplan-style"), (0.50, 0.50, "Chinchilla-style")]:
    N, D = compute_optimal(100 * C_CHINCHILLA, a, b)
    print(f"{label:>17}: 100x compute -> N = {N:.2e}, D = {D:.2e} "
          f"({D / N:.1f} tokens/param)")

Running this prints:

anchor: N = 70.0e9, D = 1.40e12  =>  C = 6*N*D = 5.88e+23 FLOPs
   C (FLOPs)   N_opt (params)   D_opt (tokens)  tokens/param
       1e+19        2.887e+08        5.774e+09          20.0
       1e+20        9.129e+08        1.826e+10          20.0
       1e+21        2.887e+09        5.774e+10          20.0
       1e+22        9.129e+09        1.826e+11          20.0
       1e+23        2.887e+10        5.774e+11          20.0
       1e+24        9.129e+10        1.826e+12          20.0
       1e+25        2.887e+11        5.774e+12          20.0
     Kaplan-style: 100x compute -> N = 2.02e+12, D = 4.85e+12 (2.4 tokens/param)
 Chinchilla-style: 100x compute -> N = 7.00e+11, D = 1.40e+13 (20.0 tokens/param)

With equal exponents the tokens-per-parameter ratio stays constant at the anchor’s 20:1, which is exactly why the rule of thumb is a ratio. The comparison at the bottom makes the disagreement tangible: given 100x Chinchilla’s budget, Kaplan-style allocation builds a 2-trillion-parameter model fed 2.4 tokens per parameter, while Chinchilla-style builds a model a third that size trained on nearly 3x the tokens. These numbers are illustrations of the published exponents, anchored to one published data point — not new measurements.

Tokens per parameter in the wild

If 20 tokens per parameter is optimal, why did the compute-accounting table earlier show 7B models trained at 143 and 286 tokens per parameter? Work the arithmetic from the deployment side. The Chinchilla rule minimizes loss for a fixed training budget: it says nothing about the cost of using the model afterward, which is roughly \(2N\) FLOPs per generated token, forever, for every user. Suppose you can afford to train with \(C = 8.4 \times 10^{22}\) FLOPs (the Llama 2-7B row). Chinchilla-optimal allocation would build roughly a 26B-parameter model on about 530B tokens (\(\sqrt{C / (6 \cdot 20)}\) gives \(N \approx 2.6 \times 10^{10}\), and \(D = 20N\)). The Llama 2 authors instead spent the same compute training a 7B model on 2 trillion tokens. The 7B model ends with somewhat higher loss than the hypothetical 26B one (that is the cost), but it is 3.7x cheaper to serve, fits on a single consumer GPU, and every downstream fine-tune inherits those economics. The first LLaMA paper (Touvron et al., 2023) states this reasoning explicitly: for a target quality level, they prefer the smallest model that reaches it, trained longer than “optimal,” because inference cost dominates total cost at deployment scale. Later open models pushed the ratio further still — Meta’s Llama 3 report describes pretraining its models on roughly 15 trillion tokens, which for the 8B variant is nearly 1,900 tokens per parameter, almost two orders of magnitude past the Chinchilla point. “Overtraining” relative to the compute-optimal frontier is not a mistake; it is a different objective function.

This is also where the open-weights-versus-closed distinction becomes scientifically relevant, not just commercially. Terminology first, because it is commonly garbled: a closed model is reachable only through an API — you can measure its behavior but not inspect it. An open-weights model publishes its parameters, so you can run, fine-tune, and dissect it, but not necessarily reproduce it; the LLaMA-family releases, for example, describe their data pipelines only in outline, and the training corpus itself is not distributed. A fully open-source release in the stricter sense (weights plus training data plus code plus intermediate checkpoints) is rarer; projects like EleutherAI’s Pythia suite and AI2’s OLMo aim at exactly this, so that claims about pretraining (scaling behavior, memorization, when capabilities appear during training) can be checked by outsiders. For the scientific questions this section covers, that distinction is load-bearing. A scaling-law paper about closed models cannot be independently re-fit; a contamination audit needs the corpus; a claim that some capability “emerges at \(10^{23}\) FLOPs” is untestable without checkpoints along the way. When you read model announcements, it is worth asking which of the three tiers you are looking at, and which claims are checkable at that tier.

Emergence, or a trick of the ruler?

As models scaled, researchers noticed that some capabilities seem to appear suddenly. Wei et al. (2022, “Emergent Abilities of Large Language Models”) collected examples: on tasks like multi-step arithmetic and certain BIG-Bench problems, models perform at chance across several orders of magnitude of scale and then jump to well above chance at some threshold. They defined an ability as emergent if it is absent in smaller models and present in larger ones, making it, under that definition, unpredictable from small-scale extrapolation. The claim mattered practically (it suggests you cannot forecast what the next model will do) and rhetorically (it fed both hype and safety concerns).

Schaeffer, Miranda, and Koyejo (2023, “Are Emergent Abilities of Large Language Models a Mirage?”) pushed back. Their argument: most claimed emergence appears when performance is measured with nonlinear or discontinuous metrics, such as exact-match accuracy on a multi-digit answer, where being almost right scores zero. The model’s underlying per-token log-likelihood typically improves smoothly with scale; the metric manufactures the cliff. When they re-scored several tasks with smoother metrics (e.g., token-level edit distance), the reported jumps largely turned into gradual curves, and they showed they could induce “emergence” in ordinary autoencoders just by choosing a harsh metric.

Where does that leave us? The honest summary: smooth loss curves and sharp task-level jumps are not contradictory (a task that requires many things to go right at once can flip quickly even when each ingredient improves gradually), and whether “emergence” names a real phenomenon or a measurement artifact partly depends on which quantity you care about. If you deploy a system that is graded exact-match, the cliff is real for you. The debate is a standing reminder to ask, of every capability claim: measured how, on what metric, with how many models between the last “can’t” and the first “can”?

Pitfall. Do not read scaling laws as guarantees about your training run. The published fits assume tuned hyperparameters at every scale, a data distribution like the one measured, and single-epoch training on effectively unlimited data. A recurring mistake is extrapolating a loss curve from three small runs (too few points, one seed), or comparing models trained with different schedules, one of the setup differences behind the gap between Kaplan’s exponents and Chinchilla’s. Fit trends on many runs, report the fitting range, and never extrapolate far beyond it.

Going deeper

The clean story above (one objective, one power law, one allocation rule) leaves out several threads that are active research.

The loss is not the product. Everything in this section optimizes and measures next-token cross-entropy, but nobody deploys a model to minimize cross-entropy. The link between pretraining loss and downstream capability is empirical, reasonably strong on average, and unreliable in the particulars: two models with equal validation loss can differ noticeably on reasoning or coding benchmarks, especially when their data mixtures differ. Fitting scaling laws directly to downstream metrics is harder (those metrics are noisier and often bounded), and this gap is one reason model developers increasingly report suites of task evaluations alongside loss.

Data constraints bend the laws. Chinchilla-style analysis assumes fresh data is unlimited. Muennighoff et al. found that, in their tested data-constrained settings and compute ranges, repeating data for several epochs incurred relatively little penalty before diminishing returns became important. Treat that as an empirical result with a domain of validity, not a universal allowance for “free” epochs.

Architecture can decouple N from compute. The \(6ND\) accounting assumes every parameter participates in every token. Mixture-of-experts (MoE) models break this: layers hold many parallel “expert” MLPs, and a small learned router activates only one or a few experts per token, so parameter count and per-token FLOPs become separately tunable dials. The Switch Transformer paper (Fedus et al., 2022) reports training trillion-parameter sparse models at the per-token cost of far smaller dense ones. Scaling laws for sparse models need at least three variables (total parameters, active parameters, data), and published fits are correspondingly younger and less settled than the dense ones.

The frontier of the fit keeps moving. Precision changes the arithmetic (bf16 versus fp8 training changes what a FLOP buys); inference-time compute (spending more forward passes per query via longer reasoning chains or search) adds a new axis that training-time scaling laws do not cover; and distillation lets small models inherit capability from large ones, muddying what “a model trained with \(C\) FLOPs” even means. None of this repeals the empirical regularities you measured in the toy experiment; all of it complicates their interpretation at the frontier.

Common pitfalls

  • Symptom: your “scaling study” shows the bigger model doing worse. Cause: one learning rate reused across sizes — large models often need different (usually smaller) learning rates, and short budgets penalize them. Fix: tune the learning rate at each size (even a small sweep), and compare at equal compute, not equal steps.
  • Symptom: validation loss looks great, but generated text is memorized chunks of training data. Cause: duplicated documents in the corpus; the model is rewarded for memorizing them. Fix: deduplicate before training and check that train and validation sets share no near-duplicates.
  • Symptom: a capability “emerges” in your evaluation between two model sizes. Cause: possibly a discontinuous metric (exact match) over a smoothly improving quantity. Fix: also plot a smooth quantity (log-likelihood of the correct answer, partial credit) across the same sizes before claiming a discontinuity.
  • Symptom: the model sized by the 20-tokens-per-parameter rule is too costly to serve: a smaller model trained on more tokens would cost less for every generated token. Cause: the rule optimizes training compute only; serving cost and latency favor smaller, longer-trained models. Fix: decide the deployment budget first, then choose the largest model you can afford to serve, and train it past the compute-optimal point.

Try it yourself

  1. Add a fourth model size (d = 512) to the scaling script and a steps=1200 run for each size. Does the loss gap between the two largest models grow or shrink with more steps? Time your runs.
  2. Fit a line to log(loss - L_floor) versus log(N) for your runs, trying a few values of the irreducible-loss floor L_floor. How sensitive is the fitted exponent to that choice? (This sensitivity is a real issue in the published papers.)
  3. Vary data instead of size: train the d = 32 model on 1%, 10%, and 100% of the corpus (same steps). Which regime (data-limited or capacity-limited) does each run land in?
  4. In the compute calculator, add an “inference-aware” column: fix N at half the compute-optimal value and solve for the D that uses the same total compute. How many tokens per parameter is that?

Further reading

  • Kaplan et al., “Scaling Laws for Neural Language Models,” 2020. arxiv.org/abs/2001.08361 — the original power-law fits; read Section 3 for the experimental design that Chinchilla later revised.
  • Hoffmann et al., “Training Compute-Optimal Large Language Models,” NeurIPS 2022. arxiv.org/abs/2203.15556 — the Chinchilla paper; its three estimation approaches are a model of triangulating an empirical claim.
  • Porian et al., “Resolving Discrepancies in Compute-Optimal Scaling of Language Models,” NeurIPS 2024. arxiv.org/abs/2406.19146 — the postmortem that pins the Kaplan-Chinchilla gap on three concrete methodological factors.
  • Brown et al., “Language Models are Few-Shot Learners” (GPT-3), NeurIPS 2020. arxiv.org/abs/2005.14165 — Section 2.2 and Table 2.2 document the data mixture and epoch-weighting choices discussed here.
  • Lee et al., “Deduplicating Training Data Makes Language Models Better,” ACL 2022. arxiv.org/abs/2107.06499 — the dedup audit behind our MinHash example, with memorization measurements.
  • Muennighoff et al., “Scaling Data-Constrained Language Models,” NeurIPS 2023. arxiv.org/abs/2305.16264 — what happens to scaling laws when you run out of fresh data and must repeat epochs.
  • Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” 2023. arxiv.org/abs/2302.13971 — the clearest published statement of the inference-cost argument for training small models past compute-optimal.
  • Wei et al., “Emergent Abilities of Large Language Models,” TMLR 2022. arxiv.org/abs/2206.07682 — the paper that defined and catalogued claimed emergent abilities.
  • Schaeffer, Miranda, Koyejo, “Are Emergent Abilities of Large Language Models a Mirage?” NeurIPS 2023. arxiv.org/abs/2304.15004 — the metric-choice rebuttal; read it side by side with Wei et al. and form your own view.
  • d2l.ai, “Large-Scale Pretraining with Transformers.” d2l.ai/chapter_attention-mechanisms-and-transformers/large-pretraining-transformers.html — a runnable textbook treatment of the pretraining objectives in this chapter.

Before class you should be able to

  • State the pretraining objective and why it is called self-supervised
  • List the main stages of a data pipeline and explain why deduplication matters twice (memorization, contamination)
  • Derive the \(C \approx 6ND\) estimate and say when it breaks down
  • State the Chinchilla rule of thumb and what quantity it optimizes
  • Explain why training a smaller model past compute-optimal can be the right call
  • Summarize the emergence debate: what the claimed cliffs are, and the metric-choice rebuttal
  • Read a small scaling script and say what its comparison does and does not establish

Connects to

  • HW2 due today · HW3 out today (Transformer from scratch, due at S22) — HW3’s training step is this section’s loop in miniature (causal mask, cross-entropy at every position), on a toy copy task, and the toy scaling script is a template for small controlled comparisons.
  • This session is covered by Quiz 3 (Module 3: S13–S22), given at S23.
  • The compute and data-quality themes return in S26 (efficiency) and S27 (evaluation).

← S17 · Transformer II · All readings · S19 · Fine-Tuning & LoRA →