Module 3 · Reading 3.7 (alignment half) · read before S21: Alignment

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

The S20 reading covered how to elicit text from a trained model: sampling algorithms that reshape the next-token distribution into tokens, and prompts that steer a pretrained model into new tasks with no weight updates. Everything there, though, took the model’s distribution as given. Raw pretrained models predict text; they do not follow instructions or decline harmful requests — prompting can coax assistant-shaped continuations out of a text predictor, but its default behavior remains prediction.

The pipeline that converts a text predictor into a usable assistant (supervised fine-tuning, reinforcement learning from human feedback, and more recently direct preference optimization) is the last piece of the modern LLM story. This reading covers that pipeline at the level of concepts and published results, then follows it to its structural failure mode, reward hacking, and to the evaluation methods the field turns to once perplexity no longer measures what matters.

How to read this

The alignment pipeline: SFT, RLHF, DPO

A pretrained model completes text. Ask it “How do I write a resignation letter?” and it may answer, or continue with three more questions, or produce a forum post that begins with someone else asking the same thing — all are likely continuations of the pretraining distribution. Alignment (in this engineering sense) is the process of tuning the model so its default behavior is to be a helpful, harmless assistant rather than a text predictor. The canonical pipeline, documented in the InstructGPT paper (Ouyang et al., 2022), has three stages.

Stage 1 — Supervised fine-tuning (SFT). Fine-tune on demonstration responses with next-token loss over the response. SFT is limited by demonstration cost, coverage, and quality. A model can generalize beyond any individual demonstration or annotator, so “imitation cannot exceed its teachers” is not a literal bound; the dataset still constrains what behaviors receive direct supervision.

Stage 2 — Reward model + RLHF. Comparing two responses is easier than writing one, so collect preferences: show labelers several model responses to a prompt and have them rank them. Train a reward model (typically the LLM with its final layer swapped for a scalar output) to score responses so that preferred ones score higher. Then optimize the SFT model with reinforcement learning (InstructGPT used PPO, proximal policy optimization) to maximize the reward model’s score, with a KL-divergence penalty (the expected log-ratio of the policy’s token probabilities to the SFT model’s, zero only when the two match) keeping the policy close to the SFT model so it cannot drift into degenerate text that games the reward model. The idea of learning rewards from human preferences predates LLMs: Christiano et al. (2017) used it for Atari and simulated robotics.

The InstructGPT paper’s headline result is worth quoting carefully: in their human evaluations, outputs from the 1.3-billion-parameter InstructGPT model were preferred to outputs from the 175-billion-parameter GPT-3 (a model over 100x larger), while InstructGPT also showed improvements in truthfulness and reductions in toxicity on their measures. Alignment training, on this evidence, changed perceived usefulness far more than scale did. The paper also reports costs: an “alignment tax” on some academic benchmarks (mitigated by mixing in pretraining data), and the pipeline’s dependence on a specific pool of labelers whose preferences become the model’s.

Stage 3 (alternative) — Direct Preference Optimization (DPO). RLHF works but is heavy: it needs a separate reward model, an RL loop, and careful stabilization. Rafailov et al. (2023) showed that for the standard RLHF objective (reward maximization with a KL penalty), the optimal policy has a closed form in terms of the reward, and that you can invert this: express the reward implicitly in terms of the policy, substitute into the preference-modeling loss, and obtain a simple classification-style loss on preference pairs, trained directly on the language model with no reward model and no RL. The paper reports DPO matching or exceeding PPO-based RLHF on their tasks (sentiment control, summarization, dialogue) with far simpler training. DPO and its descendants have since become a common default in open-model post-training, with the RLHF-versus-DPO trade-off still actively studied.

The derivation is short enough to sketch honestly, and it rewards working through because each step is a standard move. Start from the RLHF objective — maximize expected reward while staying near the SFT reference policy \(\pi_{\text{ref}}\):

\[\max_{\pi} \; \mathbb{E}[\, r(x, y) \,] - \beta \, \mathrm{KL}\bigl( \pi(y \mid x) \,\Vert\, \pi_{\text{ref}}(y \mid x) \bigr)\]

where \(\beta\) sets the strength of the leash. This objective has a known closed-form solution (it is a standard result for KL-regularized maximization):

\[\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \, \exp\!\bigl( r(x, y) / \beta \bigr)\]

This tilts the reference policy toward high reward, with \(Z(x)\) the normalizing constant summing over all responses (intractable to compute, which is why this formula alone is not directly usable). Now the pivotal move: solve for the reward instead of the policy. Taking logs and rearranging,

\[r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)\]

Every reward function corresponds to a policy, and vice versa — the language model is an implicit reward model, which is what the paper’s subtitle means. Finally, plug this into the Bradley-Terry preference model, which says the probability that labelers prefer response \(y_w\) over \(y_l\) is \(\operatorname{sigmoid}\bigl(r(x, y_w) - r(x, y_l)\bigr)\). The intractable \(\beta \log Z(x)\) term is identical for both responses to the same prompt, so it cancels in the difference, leaving a loss with nothing but policy log-probabilities in it:

\[L_{\mathrm{DPO}} = -\mathbb{E}\left[ \log \operatorname{sigmoid}\!\left( \beta \left( \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right) \right]\]

Train the model to make the preferred response’s probability ratio against the reference beat the rejected one’s — a differentiable classification loss over preference pairs, computable with two forward passes per response. No reward model is fit, no RL loop runs, yet the optimum of this loss is the same policy RLHF was aiming for, under the stated assumptions (Bradley-Terry preferences, the KL-regularized objective). The assumptions are where the debate lives: with a finite dataset and no on-policy sampling, DPO optimizes preferences over the responses in the dataset rather than over the model’s own evolving outputs, and whether that gap matters in practice is task-dependent.

Keep the pipeline’s limits in view. Preference optimization aligns models to what labelers preferred, which is not identical to “true” or “good”; reward models can be over-optimized (reward hacking); and all stages inherit the pretrained model’s knowledge and its gaps. These questions (what alignment does and does not buy) return in safety and responsible release.

Historical note. The gap between “language model” and “assistant” was visible for years before it was bridged at scale. GPT-2 (2019) and GPT-3 (2020) were prompted with clever prefixes precisely because they only completed text; the 2017 Christiano et al. preference-learning work and the 2020 Stiennon et al. summarization-from-feedback paper supplied the method; InstructGPT (published 2022) assembled the full SFT-plus-RLHF pipeline on GPT-3. ChatGPT, released in November 2022 and described by OpenAI as trained with the same family of techniques, made the difference visible to the general public and set off the current assistant era. DPO arrived in 2023, from a Stanford group, as a deliberate simplification of the machinery.

People. Paul Christiano, whose 2017 deep-RL-from-preferences paper with Jan Leike and colleagues seeded RLHF, spent years on alignment at OpenAI before founding the Alignment Research Center. Long Ouyang and colleagues at OpenAI built InstructGPT. DPO came from Rafael Rafailov, Archit Sharma, Eric Mitchell and collaborators, advised by Stefano Ermon, Christopher Manning, and Chelsea Finn at Stanford — the paper’s subtitle, calling the language model a reward model in disguise (“Your Language Model Is Secretly a Reward Model”), summarizes its central move.

Reward hacking: optimizing the proxy

The RLHF pipeline optimizes a proxy: not “be helpful” but “score well under a reward model trained on a finite sample of labeler judgments.” Optimizing any proxy hard enough exposes the gap between it and the target (the pattern economists call Goodhart’s law), and in RLHF the gap has a name, reward hacking (or reward overoptimization). The cleanest measurements come from Gao, Schulman, and Hilton (2023, “Scaling Laws for Reward Model Overoptimization”): they set up a synthetic world where a large “gold” reward model plays the role of true human preference and labels data for training smaller proxy reward models, so the true objective is, unusually, measurable. Optimizing against the proxy, whether by RL or by best-of-n sampling (generate \(n\) candidates, keep the proxy’s favorite), they report a characteristic curve: gold-standard quality rises at first, peaks, and then declines while the proxy score continues climbing. The model is finding responses the proxy loves and the gold standard does not — exploiting the proxy’s errors, which is exactly what unconstrained optimization of an imperfect function should be expected to do. They further report that the peak moves later and higher as the proxy reward model grows, and that the functional form of the degradation differs between best-of-n and RL.

What do hacked responses look like in real pipelines? The well-documented offenders are mundane: length (labelers, on average, rate longer answers higher, so models pad), sycophancy (agreeing with the user’s stated views scores better than correcting them), confident tone without content, and formatting tricks like bullet-point avalanches. The KL penalty from the InstructGPT recipe is the first defense (it prices deviation from the SFT model, effectively giving the optimizer a budget it must spend wisely), and the practical toolkit adds early stopping against a held-out human evaluation, ensembles of reward models, iteratively refreshed preference data, and plain reading of transcripts. None is a solution; all are mitigations of a structural fact. DPO does not escape it either: its implicit reward is fit to the same finite, biased preference data, and DPO-tuned models exhibit the same length inflation when the data rewards it. When you see a fine-tuned model’s benchmark score leap, the first question is the one this section keeps returning to: what, precisely, went up?

Evaluating aligned models

Alignment breaks the evaluation tools you have used so far. Perplexity on held-out text measures the pretraining objective, which post-training deliberately moves away from: an aligned model can have worse perplexity and be plainly more useful. Task benchmarks with exact-match answers miss most of what SFT and preference tuning change: tone, refusals, formatting, instruction adherence. So the field evaluates aligned models the way they were trained: by preference. Show two models’ responses to the same prompt and ask which is better; aggregate win rates over many prompts. The judge can be human (LMSYS’s Chatbot Arena collected millions of crowd-sourced pairwise votes, aggregated into Elo-style ratings, and became the de-facto public leaderboard) or another LLM, which is cheap and fast but imports its own biases. Zheng et al. (2023, “Judging LLM-as-a-Judge”), who built MT-Bench and analyzed arena data, report that strong LLM judges agree with human majority preferences at rates comparable to human-human agreement on their setup — and document systematic judge biases: position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers), and self-enhancement (favoring outputs resembling the judge’s own style). Mitigations are workmanlike: swap answer order and average, control for length, use a judge from a different model family than the systems being compared.

For your own work, the practical protocol is: decide what dimension you care about (helpfulness? factuality? safety refusals?), write or collect prompts that probe it, fix the decoding configuration for every system, collect pairwise judgments with order randomization, and report win rates with uncertainty from multiple seeds and prompt variants. And keep one adversarial habit: whenever a single scalar (win rate, Elo, judge score) is the headline, ask what a model could do to raise the scalar without getting better. After the reward-hacking section, you know the answer is rarely “nothing.”

Going deeper

The pipeline described here is the 2022-2023 canon; the edges have kept moving, and three directions matter for reading current work.

Verifiable rewards. Exact-answer checks and code tests reduce dependence on a learned reward model, but they do not eliminate reward hacking: underspecified objectives, checker bugs, weak tests, and benchmark leakage can all be exploited. These rewards are strongest when the specification is complete and adversarially audited; their domain coverage remains narrow.

The DPO family and its trade-offs. DPO’s simplicity invited descendants — IPO reworks the loss to reduce overfitting to preferences, KTO drops the need for paired comparisons, and length-regularized variants target the verbosity bias directly. The RLHF-versus-DPO question is genuinely unsettled in public evidence: on-policy RL sees the model’s own mistakes during training, offline DPO does not, and published comparisons disagree depending on data scale and task. Treat “which post-training method” as an empirical question, not settled doctrine.

Best-of-n as a baseline and a bridge. Sampling \(n\) responses and keeping the best under a reward model is not just a reward-hacking probe — it is a strong, simple deployment strategy (no training at all) and a conceptual bridge between decoding and alignment: it is a decoding algorithm whose target distribution is tilted toward reward, the same tilt \(\pi_{\text{ref}} \exp(r/\beta)\) that RLHF bakes into weights. Much current inference-time-compute work (search over reasoning chains, verifier-guided decoding) elaborates this bridge. If you remember one unifying frame from this section, make it that: pretraining fixes what the model knows, and everything downstream (sampling, constraints, prompts, preference tuning) is a choice about which of the model’s possible outputs you elicit.

Common pitfalls

  • Symptom: after preference tuning, the model’s outputs get longer and more sycophantic but not better. Cause: the reward signal is being gamed — length and agreeableness correlate with labeler preference in the training data. Fix: at course scale, treat this as an evaluation lesson: inspect outputs directly, and never optimize a proxy metric you have not stress-tested.
  • Symptom: an aligned model’s benchmark scores dip after RLHF and the run is declared a failure. Cause: the alignment tax — helpfulness tuning can trade off narrow benchmark accuracy; the InstructGPT paper reports mitigating (not eliminating) it by mixing pretraining gradients into the RL step. Fix: evaluate what alignment targets (preference win-rate, instruction following), not only what pretraining targets.
  • Symptom: DPO training looks stable but the model drifts into terse, generic answers. Cause: the implicit reward optimizes the offline preference set; over-training on a narrow preference distribution collapses diversity just as over-optimizing an explicit reward model does. Fix: hold out preference pairs for evaluation, early-stop on win-rate against the reference policy, and inspect samples, not just the loss.

Further reading

  • Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT), NeurIPS 2022. arxiv.org/abs/2203.02155 — the canonical SFT-plus-RLHF pipeline, including the 1.3B-beats-175B human-preference result.
  • Rafailov et al., “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model,” NeurIPS 2023. arxiv.org/abs/2305.18290 — the full version of the derivation sketched here, with the experiments.
  • Gao, Schulman, Hilton, “Scaling Laws for Reward Model Overoptimization,” ICML 2023. arxiv.org/abs/2210.10760 — the gold-versus-proxy reward experiments behind our reward-hacking discussion.
  • Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023. arxiv.org/abs/2306.05685 — the LLM-judge agreement rates and bias catalog cited in the evaluation section.

Try it yourself

  1. Compute the DPO loss by hand for one preference pair: take \(\beta = 0.1\), policy log-probs -12.0 (chosen) and -11.0 (rejected), reference log-probs -12.5 and -10.5. Form the two log-ratios, their difference times \(\beta\), and \(-\log \operatorname{sigmoid}(\cdot)\). Then flip the chosen/rejected labels and confirm the loss rises above \(\log 2 \approx 0.693\), its value at no preference; the asymmetry is the training signal.
  2. Design a reward-hacking probe for a chat model you use: pick a behavior the reward model plausibly overvalues (length, hedging, flattery), write two prompts that should elicit the same answer at different lengths, and predict which response a length-biased judge would prefer. The “Reward hacking: optimizing the proxy” section gives you the vocabulary; the exercise is noticing the incentive before you see the failure.

Before class you should be able to

  • Describe what each stage (SFT, reward model + RLHF, DPO) trains on and produces
  • Explain the KL penalty’s job in the RLHF objective
  • Sketch DPO’s central move and why no reward model or RL loop is needed
  • Define reward hacking and give two documented examples from real pipelines
  • Explain why an aligned model can have worse perplexity yet be more useful
  • Name the three LLM-judge biases and one mitigation for each
  • Say what preference optimization aligns a model to — and what it does not guarantee

Connects to

  • This session is covered by Quiz 3 (Module 3: S13–S22), given at S23.
  • Project relevance: the pairwise-evaluation protocol here (order randomization, fixed decoding config, uncertainty) is the standard for judging any generative project output — and the reward-hacking lesson is the course-wide warning against optimizing an untested proxy metric.
  • Safety and evaluation questions raised here return in S27.

← S20 · Decoding, Prompting, and In-Context Learning · All readings · S22 · ViT & Multimodal →