S22 Reading — ViT & Multimodal | Deep Learning

Module 3 · Reading 3.8 · read before S22: ViT & Multimodal

The PDF posted in Canvas is the assigned version of this reading. It is built from this page, so the text is identical; anything here marked as deep reading is optional and is not in the PDF.

For most of the 2010s, computer vision and natural language processing ran on different engines: convolutions for pixels, recurrence and then attention for words. The Transformer ended the split. The Vision Transformer (ViT) showed that if you chop an image into patches and feed them to a plain Transformer as if they were tokens, you can match and exceed the best CNNs — provided you have enough data, a caveat that turns out to be the scientifically interesting part. And once images and text live in the same architecture, it is a short step to putting them in the same representation space: CLIP trains an image encoder and a text encoder so that matching image-caption pairs land close together, which yields classifiers you can define at test time by typing the class names.

This section builds both core mechanisms in code: the patch embedding that turns an image into a token sequence, and the symmetric contrastive loss that binds two modalities together.

An image is a sequence of patches

The obstacle to running a Transformer on pixels is quadratic cost: self-attention over a 224x224 image treated as 50,176 pixel-tokens is far too expensive. The ViT paper (Dosovitskiy et al., “An Image is Worth 16x16 Words,” published at ICLR 2021) resolved it with a blunt instrument: cut the image into non-overlapping 16x16 patches, flatten each patch into a vector, and project it linearly into the model dimension. A 224x224 image becomes a sequence of \((224/16)^2 = 196\) patch tokens, a perfectly ordinary sequence length. Add learned position embeddings (the Transformer is otherwise permutation-invariant and would not know where each patch came from), prepend a learnable [CLS] token whose final representation serves as the image summary for classification, and run a standard Transformer encoder. No convolutions, no pooling pyramids, no architectural concessions to vision beyond the patch cut itself.

The patch embedding is the only new mechanism, and it is worth implementing exactly, because getting the reshape right is where implementations go wrong.

import torch
import torch.nn as nn

torch.manual_seed(0)

B, C, H, W = 2, 3, 32, 32       # batch of 2 RGB images, 32x32
P = 4                            # patch size
D = 96                           # embedding dimension
n_patches = (H // P) * (W // P)  # 8 * 8 = 64 patches per image

imgs = torch.randn(B, C, H, W)

def patchify(x, p):
    """(B, C, H, W) -> (B, n_patches, p*p*C), row-major patch order."""
    B, C, H, W = x.shape
    x = x.reshape(B, C, H // p, p, W // p, p)   # split H and W into blocks
    x = x.permute(0, 2, 4, 1, 3, 5)             # (B, H/p, W/p, C, p, p)
    return x.reshape(B, (H // p) * (W // p), C * p * p)

def unpatchify(patches, p, c, h, w):
    """Inverse of patchify."""
    B = patches.shape[0]
    x = patches.reshape(B, h // p, w // p, c, p, p)
    x = x.permute(0, 3, 1, 4, 2, 5)
    return x.reshape(B, c, h, w)

patches = patchify(imgs, P)
print(f"images  : {tuple(imgs.shape)}")
print(f"patches : {tuple(patches.shape)}   (expect ({B}, {n_patches}, {C * P * P}))")

# round-trip check: patchify must lose no information
recon = unpatchify(patches, P, C, H, W)
print(f"round-trip max |x - unpatchify(patchify(x))| = {(imgs - recon).abs().max().item():.1f}")

# linear projection to embeddings + [CLS] token + position embeddings
proj = nn.Linear(C * P * P, D)
cls_token = nn.Parameter(torch.zeros(1, 1, D))
pos_embed = nn.Parameter(torch.randn(1, n_patches + 1, D) * 0.02)

tokens = proj(patches)                              # (B, 64, 96)
tokens = torch.cat([cls_token.expand(B, -1, -1), tokens], dim=1)
tokens = tokens + pos_embed
print(f"tokens  : {tuple(tokens.shape)}   (expect ({B}, {n_patches + 1}, {D}))")

# the same projection as a strided convolution (the trick real ViTs use)
conv = nn.Conv2d(C, D, kernel_size=P, stride=P)
with torch.no_grad():
    conv.weight.copy_(proj.weight.reshape(D, C, P, P))
    conv.bias.copy_(proj.bias)
conv_tokens = conv(imgs).flatten(2).transpose(1, 2)  # (B, 64, 96)
diff = (conv_tokens - proj(patches)).abs().max()
print(f"max |Linear-on-patches - strided Conv2d| = {diff.item():.2e}")

Running this prints:

images  : (2, 3, 32, 32)
patches : (2, 64, 48)   (expect (2, 64, 48))
round-trip max |x - unpatchify(patchify(x))| = 0.0
tokens  : (2, 65, 96)   (expect (2, 65, 96))
max |Linear-on-patches - strided Conv2d| = 0.00e+00

Three verified facts. The shapes work out: 64 patches of dimension 4*4*3 = 48, becoming 65 tokens of dimension 96 after projection, [CLS], and position embeddings. The round-trip is exact: unpatchify(patchify(x)) reproduces the input with maximum error 0.0, confirming patchify is pure rearrangement, losing nothing. And the linear-projection-per-patch is identical (0.00e+00) to a Conv2d with kernel size and stride equal to the patch size — which is both how production ViTs implement patch embedding and a nice conceptual bridge: the ViT does contain exactly one convolution’s worth of convolution, applied with no overlap.

Inductive bias versus data scale

A CNN hard-codes strong locality and translation-equivariance priors. A ViT has fewer vision-specific priors, but not none: patch projection and Transformer weights are shared across positions, self-attention without position is permutation equivariant, and the patch grid/position encoding supplies structure. Local interactions are available but not forced. Larger data and stronger augmentation can reduce the value of the convolutional prior; that does not prove priors universally become constraints.

The ViT paper reports lower accuracy than comparable ResNets when trained on ImageNet-1k alone, roughly competitive results with ImageNet-21k pretraining, and stronger results for large ViTs pretrained on JFT-300M. These experiments show that data scale and training recipe can compensate for weaker vision-specific priors; they do not establish that convolutional priors themselves become a universal constraint. DeiT later showed that augmentation, regularization, and distillation move the small-data boundary substantially.

It is worth connecting this to the scaling-laws story: in both vision and language, the 2020-2021 lesson was that a generic architecture plus more data beats a specialized architecture at sufficient scale. That is an empirical regularity with a domain of validity, not a theorem — for small datasets (most scientific and medical imaging, most course projects), CNN-style biases still earn their keep.

Historical note. Applying attention to images was tried repeatedly before ViT — hybrids that added attention layers to CNNs, models that restricted attention to local neighborhoods, even pixel-level Transformers on tiny images (Image GPT, 2020). What distinguished the ViT paper was the demonstration that an essentially unmodified Transformer encoder works at scale, if the input is patchified and pretraining data is plentiful. Reception was fast: within about two years of the October 2020 preprint, ViT variants were standard backbones across classification, detection, and segmentation, and the patch-embedding idea reappeared in masked-autoencoder pretraining (He et al., 2022) and diffusion model backbones (DiT, 2023).

The bias experiment on your CPU

The data-versus-bias trade-off is not reserved for Google-scale experiments; you can stage a miniature version and watch it happen. The task below is engineered so that the CNN’s priors are exactly right: each 16x16 image hides a 3x3 shape (a plus or an X) at a random position under Gaussian noise, and the class depends only on that local pattern, wherever it sits. A convolution slides the same filter everywhere, so translation equivariance and locality are precisely the structure of the problem. The ViT gets no such head start: its patch tokens carve the image into a fixed 4x4 grid (a shape can straddle up to four patches), position embeddings start random, and attention must learn that nearby pixels matter. Both models train with the same optimizer, learning rate, batch size, and step budget, at two training-set sizes.

# Inductive bias vs data scale, in miniature: a small CNN and a small ViT
# on the same local-pattern task, same optimizer, same step budget, at two
# training-set sizes. The task: is the 3x3 shape hidden in the noise a
# "plus" or an "X"? Locality and translation equivariance should help.
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

PLUS = torch.tensor([[0., 1, 0], [1, 1, 1], [0, 1, 0]]) * 2.0
X    = torch.tensor([[1., 0, 1], [0, 1, 0], [1, 0, 1]]) * 2.0

def make_data(n, gen):
    imgs = 0.3 * torch.randn(n, 1, 16, 16, generator=gen)
    labels = torch.randint(0, 2, (n,), generator=gen)
    pos = torch.randint(0, 13, (n, 2), generator=gen)   # top-left of the 3x3 patch
    for i in range(n):
        r, c = pos[i]
        imgs[i, 0, r:r + 3, c:c + 3] += PLUS if labels[i] == 0 else X
    return imgs, labels

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 2))

    def forward(self, x):
        return self.net(x)

class SmallViT(nn.Module):
    def __init__(self, d=48, n_head=4, n_layer=2, p=4):
        super().__init__()
        self.embed = nn.Conv2d(1, d, kernel_size=p, stride=p)   # patchify
        self.pos = nn.Parameter(torch.randn(1, (16 // p) ** 2, d) * 0.02)
        enc = nn.TransformerEncoderLayer(d, n_head, 4 * d, batch_first=True,
                                         dropout=0.0, norm_first=True)
        self.blocks = nn.TransformerEncoder(enc, n_layer, enable_nested_tensor=False)
        self.head = nn.Linear(d, 2)

    def forward(self, x):
        tokens = self.embed(x).flatten(2).transpose(1, 2) + self.pos
        return self.head(self.blocks(tokens).mean(dim=1))       # mean-pool

def train_eval(model, Xtr, ytr, Xte, yte, steps=400):
    gen = torch.Generator().manual_seed(2)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
    for step in range(steps):
        ix = torch.randint(0, len(Xtr), (64,), generator=gen)
        loss = F.cross_entropy(model(Xtr[ix]), ytr[ix])
        opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    with torch.no_grad():
        return (model(Xte).argmax(1) == yte).float().mean().item()

gen = torch.Generator().manual_seed(1)
Xte, yte = make_data(1000, gen)

print(f"{'train size':>10} {'model':>9} {'params':>8} {'test acc':>9}")
for n_train in [256, 4096]:
    Xtr, ytr = make_data(n_train, gen)
    for name, cls in [("CNN", SmallCNN), ("ViT", SmallViT)]:
        torch.manual_seed(0)                 # identical init treatment
        model = cls()
        n_params = sum(p.numel() for p in model.parameters())
        acc = train_eval(model, Xtr, ytr, Xte, yte)
        print(f"{n_train:>10} {name:>9} {n_params:>8,} {acc:>9.4f}")

Running this prints:

train size     model   params  test acc
       256       CNN    4,866    0.9960
       256       ViT   58,226    0.8790
      4096       CNN    4,866    0.9990
      4096       ViT   58,226    0.9920

The pattern is the ViT paper’s Figure 3 in miniature. With 256 training images, the CNN is already at 99.6% while the ViT (despite having 12x the parameters) sits at 87.9%: the ViT is spending its data learning what the CNN was born knowing. Multiply the data by 16 and the gap nearly closes (99.9% versus 99.2%). Honest caveats, in the spirit this course keeps insisting on: this is one seed, one learning rate shared by both architectures (a per-model sweep could shift the numbers), a tiny ViT without the augmentation recipes that make real ViTs data-efficient, and a task designed to flatter convolution. The point is not that CNNs are better — it is that the low-data penalty for missing priors is real, measurable, and shrinks with data, which is the shape of the published result. When your semester project has 500 labeled images, this table is the argument for a CNN or a pretrained ViT, never a from-scratch one.

Reading a ViT’s attention maps

One thing a ViT gives you that a CNN does not: every layer’s attention weights are an explicit, inspectable matrix. For a classifier with a [CLS] token, the natural question is which patches does the class token consult? On our synthetic task we know where the answer should be, so we can score the attention against the truth rather than just admiring it. The script trains a small [CLS]-token ViT (with a hand-written attention block that saves its weights), then checks, over 500 test images, whether the most-attended patch actually overlaps the hidden shape.

# Where does a trained ViT look? Train a tiny ViT with a [CLS] token on the
# plus-vs-X task, then read out the last block's CLS->patch attention weights
# and check whether they land on the patches that contain the shape.
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

PLUS = torch.tensor([[0., 1, 0], [1, 1, 1], [0, 1, 0]]) * 2.0
X    = torch.tensor([[1., 0, 1], [0, 1, 0], [1, 0, 1]]) * 2.0

def make_data(n, gen):
    imgs = 0.3 * torch.randn(n, 1, 16, 16, generator=gen)
    labels = torch.randint(0, 2, (n,), generator=gen)
    pos = torch.randint(0, 13, (n, 2), generator=gen)
    for i in range(n):
        r, c = pos[i]
        imgs[i, 0, r:r + 3, c:c + 3] += PLUS if labels[i] == 0 else X
    return imgs, labels, pos

class Block(nn.Module):
    """Pre-norm encoder block with attention weights exposed."""
    def __init__(self, d, n_head):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        self.n_head, self.d = n_head, d

    def forward(self, x):
        B, T, d = x.shape
        h = self.ln1(x)
        q, k, v = self.qkv(h).chunk(3, dim=-1)
        q, k, v = (t.view(B, T, self.n_head, -1).transpose(1, 2) for t in (q, k, v))
        att = (q @ k.transpose(-2, -1)) / math.sqrt(d // self.n_head)
        att = att.softmax(-1)
        self.last_att = att.detach()                 # (B, heads, T, T)
        x = x + self.proj((att @ v).transpose(1, 2).reshape(B, T, d))
        return x + self.mlp(self.ln2(x))

class ViT(nn.Module):
    def __init__(self, d=48, n_head=4, n_layer=2, p=4):
        super().__init__()
        self.embed = nn.Conv2d(1, d, kernel_size=p, stride=p)
        self.cls = nn.Parameter(torch.zeros(1, 1, d))
        self.pos = nn.Parameter(torch.randn(1, 17, d) * 0.02)
        self.blocks = nn.ModuleList([Block(d, n_head) for _ in range(n_layer)])
        self.head = nn.Linear(d, 2)

    def forward(self, x):
        t = self.embed(x).flatten(2).transpose(1, 2)          # (B, 16, d)
        t = torch.cat([self.cls.expand(len(x), -1, -1), t], 1) + self.pos
        for blk in self.blocks:
            t = blk(t)
        return self.head(t[:, 0])                             # classify from CLS

gen = torch.Generator().manual_seed(1)
Xtr, ytr, _ = make_data(4096, gen)
Xte, yte, pos_te = make_data(500, gen)

model = ViT()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
sample_gen = torch.Generator().manual_seed(2)
for step in range(400):
    ix = torch.randint(0, len(Xtr), (64,), generator=sample_gen)
    loss = F.cross_entropy(model(Xtr[ix]), ytr[ix])
    opt.zero_grad(); loss.backward(); opt.step()
model.eval()

with torch.no_grad():
    acc = (model(Xte).argmax(1) == yte).float().mean()
    # CLS row of the last block's attention, averaged over heads: (B, 17)
    cls_att = model.blocks[-1].last_att.mean(dim=1)[:, 0, :]
    patch_att = cls_att[:, 1:]                                # drop CLS->CLS
print(f"test accuracy: {acc.item():.4f}")

# does the most-attended patch overlap the 3x3 shape?
hits = 0
for i in range(len(Xte)):
    r, c = pos_te[i]
    overlapping = {(rr // 4) * 4 + (cc // 4)
                   for rr in range(r, r + 3) for cc in range(c, c + 3)}
    hits += int(patch_att[i].argmax().item() in overlapping)
print(f"most-attended patch overlaps the shape: {hits}/{len(Xte)} "
      f"({100 * hits / len(Xte):.1f}%)  [chance would be ~13%]")

# one image in detail: attention over the 4x4 patch grid
i = 0
r, c = pos_te[i]
print(f"\nimage 0: shape top-left at pixel ({r},{c}) "
      f"-> patch grid rows {r // 4}-{(r + 2) // 4}, cols {c // 4}-{(c + 2) // 4}")
print("CLS attention over patches (x100, row-major 4x4 grid):")
grid = patch_att[i].reshape(4, 4) * 100
for row in grid:
    print("  " + " ".join(f"{v:5.1f}" for v in row.tolist()))

Running this prints:

test accuracy: 0.9880
most-attended patch overlaps the shape: 483/500 (96.6%)  [chance would be ~13%]

image 0: shape top-left at pixel (12,8) -> patch grid rows 3-3, cols 2-2
CLS attention over patches (x100, row-major 4x4 grid):
    1.3   0.7   2.1   1.6
    1.0   0.7   1.0   2.5
    1.1   1.1   0.5   1.7
    1.4   1.0  74.2   4.8

The trained model’s [CLS] token concentrates its last-layer attention on the shape-bearing patch in 96.6% of test images — for image 0, 74.2% of the attention mass lands on exactly the patch (row 3, column 2) that contains the shape, with the rest spread thinly. Since a random patch choice would overlap the shape only about an eighth of the time (the 3x3 shape covers on average about 2.1 of the 16 patches), the attention map is genuinely tracking the evidence. This is the same computation behind the attention-overlay visualizations in the ViT paper and many mechanistic studies, and it inherits their caveat, which deserves its own callout.

Pitfall. Attention maps are seductive, and “the model attends to X” is routinely oversold as “the model’s decision is explained by X.” Resist that jump. Attention weights are one layer’s mixing coefficients, not a causal account: information about a patch can flow through other tokens (after one block, every token is already a mixture), heads can attend to a location without the value stream using it, and large ViTs are documented to develop high-attention “artifact” tokens in uninformative background regions (Darcet et al., 2024, propose adding register tokens specifically to absorb these). Our toy result is trustworthy only because we scored attention against a known ground truth over 500 images — if you show an attention overlay in your project, pair it with a quantitative check (occlusion, ablation, or a known-location task like this one), not just a pretty heat map.

CLIP: two encoders, one space

ViT gives images the Transformer treatment; CLIP (Contrastive Language-Image Pretraining, Radford et al., ICML 2021) connects images to language. The training data is not class labels but roughly 400 million (image, text) pairs collected from the internet: captions, alt-text, surrounding text. The model is two encoders: an image encoder (a ViT or ResNet) and a text encoder (a Transformer), each followed by a projection into a shared embedding space where vectors are normalized to unit length.

The objective is contrastive: in a batch of \(N\) pairs, embed all images and all texts, compute the \(N \times N\) matrix of cosine similarities scaled by a learnable temperature, and train so that the \(N\) matched pairs (the diagonal) score higher than the \(N^2 - N\) mismatched combinations. Concretely, each row of the similarity matrix is treated as logits for an \(N\)-way classification “which caption belongs to this image?”, each column as “which image belongs to this caption?”, and the loss is the average of the two cross-entropies: the symmetric InfoNCE loss. The following script implements it and verifies, on synthetic data with a known shared structure, that minimizing it does what we claim: matched pairs align, mismatched pairs do not.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

N, D_IMG, D_TXT, D = 64, 48, 32, 16   # 64 pairs, different raw dims, shared space

# Synthetic "images" and "captions": pair i shares a hidden concept vector z_i,
# seen through two different random linear "modalities" plus noise.
g = torch.Generator().manual_seed(1)
z = torch.randn(N, 8, generator=g)
img_map = torch.randn(8, D_IMG, generator=g)
txt_map = torch.randn(8, D_TXT, generator=g)
imgs = z @ img_map + 0.1 * torch.randn(N, D_IMG, generator=g)
txts = z @ txt_map + 0.1 * torch.randn(N, D_TXT, generator=g)

img_proj = nn.Linear(D_IMG, D)
txt_proj = nn.Linear(D_TXT, D)
# log of the learnable logit scale 1/T, like CLIP
log_temp = nn.Parameter(torch.tensor(0.0))

def clip_loss():
    ie = F.normalize(img_proj(imgs), dim=-1)      # unit-norm embeddings
    te = F.normalize(txt_proj(txts), dim=-1)
    logits = ie @ te.T * log_temp.exp()           # (N, N) similarity matrix
    labels = torch.arange(N)                      # pair i matches pair i
    loss_i = F.cross_entropy(logits, labels)      # image -> which caption?
    loss_t = F.cross_entropy(logits.T, labels)    # caption -> which image?
    return (loss_i + loss_t) / 2, ie, te

params = list(img_proj.parameters()) + list(txt_proj.parameters()) + [log_temp]
opt = torch.optim.Adam(params, lr=1e-2)

print(f"chance-level loss for batch of {N}: ln({N}) = {math.log(N):.4f}")
for step in range(301):
    loss, ie, te = clip_loss()
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 75 == 0:
        with torch.no_grad():
            sims = ie @ te.T
            matched = sims.diag().mean()
            mismatched = (sims.sum() - sims.diag().sum()) / (N * N - N)
            top1 = (sims.argmax(dim=1) == torch.arange(N)).float().mean()
        print(f"step {step:3d}  loss {loss.item():.4f}  "
              f"matched sim {matched.item():+.4f}  "
              f"mismatched sim {mismatched.item():+.4f}  "
              f"retrieval@1 {top1.item():.4f}")

Running this prints:

chance-level loss for batch of 64: ln(64) = 4.1589
step   0  loss 4.1620  matched sim +0.0298  mismatched sim -0.0012  retrieval@1 0.0000
step  75  loss 2.2355  matched sim +0.9993  mismatched sim -0.0158  retrieval@1 1.0000
step 150  loss 0.6498  matched sim +0.9995  mismatched sim -0.0157  retrieval@1 1.0000
step 225  loss 0.1396  matched sim +0.9994  mismatched sim -0.0153  retrieval@1 1.0000
step 300  loss 0.0548  matched sim +0.9993  mismatched sim -0.0142  retrieval@1 1.0000

The run confirms each claim in the design. At step 0 the loss sits at 4.1620, essentially the ln(64) = 4.1589 chance level: with random projections, the model cannot tell captions apart. As training proceeds the loss falls to 0.0548, the mean cosine similarity of matched pairs rises from 0.03 to 0.999 while mismatched pairs stay near zero, and retrieval accuracy (does image i’s nearest text embedding belong to pair i?) hits 100%. Note why the toy works so cleanly: each pair’s image and caption were generated from a shared 8-dimensional concept vector, so a linear map into 16 dimensions can align them almost perfectly. Real image-text pairs share structure far more loosely, need deep nonlinear encoders, and never reach similarity 0.999 — but the mechanics of the loss are exactly these.

Why does this produce a zero-shot classifier? Because the text encoder turns any string into a point in the shared space. To classify an image among classes you invented this morning, embed the phrases “a photo of a dog,” “a photo of a trombone,” and so on, and pick the class whose text embedding is most similar to the image embedding. The CLIP paper reports that this procedure (with no ImageNet training images used for supervision) matches the original ResNet-50’s ImageNet top-1 accuracy (the paper reports 76.2% zero-shot), and shows markedly better robustness than standard ImageNet models on distribution-shift benchmarks like ImageNet-Sketch and ImageNet-R. The paper is equally explicit about weaknesses: zero-shot CLIP performs poorly on fine-grained tasks (differentiating car models, aircraft variants), on counting, and on data truly absent from its training distribution.

Temperature, and a sigmoid alternative

Two details of the contrastive recipe deserve experiments of their own: the temperature, and the softmax itself. The temperature first. Because both embeddings are normalized to unit length, their dot product (a cosine similarity) is trapped in \([-1, 1]\). Feed that raw into a softmax over 64 candidates and the largest logit gap possible is 2, so the distribution stays stubbornly flat and cross-entropy cannot approach zero no matter how well-separated the embeddings are. Dividing by a small temperature \(T\) (equivalently, multiplying by a large scale) stretches the usable range; CLIP makes the scale a learnable parameter, initialized to 1/0.07 and capped at 100. The second experiment implements SigLIP’s alternative (Zhai et al., 2023, “Sigmoid Loss for Language Image Pre-Training”): drop the softmax entirely and treat every (image, text) cell of the similarity matrix as an independent binary classification; diagonal cells should score high, off-diagonal cells low, each under its own sigmoid, with a learnable bias handling the imbalance (in a batch of \(N\), negatives outnumber positives \(N-1\) to 1). Because no row-normalization couples the cells, the loss no longer bakes the batch size into its definition — the property the SigLIP paper exploits to decouple batch size from optimization, reporting both better small-batch behavior and stable scaling to very large batches.

# Two questions about the contrastive recipe, answered by experiment:
# (1) what does the temperature do?  (2) how does SigLIP's pairwise sigmoid
# loss compare to softmax InfoNCE when the training batch is small?
# Retrieval is always evaluated against a 256-pair held-out pool.
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

D_IMG, D_TXT, D = 48, 32, 16

def make_pairs(n, gen, img_map, txt_map):
    z = torch.randn(n, 8, generator=gen)
    imgs = z @ img_map + 2.0 * torch.randn(n, D_IMG, generator=gen)
    txts = z @ txt_map + 2.0 * torch.randn(n, D_TXT, generator=gen)
    return imgs, txts

gen = torch.Generator().manual_seed(1)
img_map = torch.randn(8, D_IMG, generator=gen)
txt_map = torch.randn(8, D_TXT, generator=gen)
tr_imgs, tr_txts = make_pairs(512, gen, img_map, txt_map)
te_imgs, te_txts = make_pairs(256, gen, img_map, txt_map)

def run(loss_type, batch_size, temp_mode="learnable", steps=400):
    torch.manual_seed(0)
    img_proj, txt_proj = nn.Linear(D_IMG, D), nn.Linear(D_TXT, D)
    log_t = nn.Parameter(torch.tensor(0.0))          # log of the logit scale 1/T (T = exp(-log_t))
    bias = nn.Parameter(torch.tensor(-10.0))         # SigLIP's learnable bias
    params = list(img_proj.parameters()) + list(txt_proj.parameters()) + [bias]
    if temp_mode == "learnable":
        params.append(log_t)
    else:                                            # fixed temperature
        with torch.no_grad():
            log_t.fill_(torch.tensor(1.0 / temp_mode).log().item())
    opt = torch.optim.Adam(params, lr=1e-2)
    bgen = torch.Generator().manual_seed(2)
    for step in range(steps):
        ix = torch.randint(0, 512, (batch_size,), generator=bgen)
        ie = F.normalize(img_proj(tr_imgs[ix]), dim=-1)
        te = F.normalize(txt_proj(tr_txts[ix]), dim=-1)
        sims = ie @ te.T * log_t.exp()
        if loss_type == "softmax":                   # symmetric InfoNCE
            labels = torch.arange(batch_size)
            loss = (F.cross_entropy(sims, labels)
                    + F.cross_entropy(sims.T, labels)) / 2
        else:                                        # SigLIP: pairwise sigmoid
            signs = 2 * torch.eye(batch_size) - 1    # +1 diagonal, -1 elsewhere
            loss = -F.logsigmoid(signs * (sims + bias)).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    final_loss = loss.item()
    with torch.no_grad():                            # retrieval over 256-pool
        ie = F.normalize(img_proj(te_imgs), dim=-1)
        te = F.normalize(txt_proj(te_txts), dim=-1)
        top1 = ((ie @ te.T).argmax(1) == torch.arange(256)).float().mean()
    return top1.item(), log_t.exp().item(), final_loss

print("part 1 - temperature (softmax InfoNCE, batch 64):")
print(f"{'temperature':>20} {'retrieval@1':>12} {'final T':>9} {'train loss':>11}")
for mode, label in [(1.0, "fixed T=1.0"), (0.07, "fixed T=0.07"),
                    ("learnable", "learnable")]:
    top1, t, fl = run("softmax", 64, mode)
    print(f"{label:>20} {top1:>12.4f} {1 / t:>9.4f} {fl:>11.4f}")

print("\npart 2 - loss type x batch size (learnable temperature):")
print(f"{'loss':>9} {'batch':>6} {'retrieval@1':>12}")
for loss_type in ["softmax", "sigmoid"]:
    for bs in [8, 64]:
        top1, _, _ = run(loss_type, bs)
        print(f"{loss_type:>9} {bs:>6} {top1:>12.4f}")

Running this prints:

part 1 - temperature (softmax InfoNCE, batch 64):
         temperature  retrieval@1   final T  train loss
         fixed T=1.0       0.6758    1.0000      3.3437
        fixed T=0.07       0.5859    0.0700      0.3118
           learnable       0.5781    0.0478      0.2264

part 2 - loss type x batch size (learnable temperature):
     loss  batch  retrieval@1
  softmax      8       0.5547
  softmax     64       0.5781
  sigmoid      8       0.6602
  sigmoid     64       0.6172

Part 1’s mechanism shows up in the train-loss column: at T = 1.0 the loss cannot fall below 3.34 (the bounded cosine logits keep the softmax flat, exactly as the arithmetic predicted), while at T = 0.07, and with the learnable scale (which settles at T ≈ 0.048, in the same region CLIP’s learned value ends up), the loss drives down to 0.2-0.3. Note what did not happen: the sharpest temperature did not win at retrieval. On this deliberately noisy toy (noise swamps signal, only 512 training pairs), the sharp runs use their extra resolution to memorize noisy training pairs, and the flat T = 1.0 run generalizes best. The published story (temperature is necessary at scale) and this toy’s story (sharpness can overfit at tiny scale) are both true, and the difference is a useful reminder that hyperparameter folklore comes with an implicit scale attached. Part 2 shows the SigLIP comparison: at batch 8, where softmax InfoNCE has only 7 negatives per positive and its \(\ln(8)\)-capped difficulty makes training signal weak, the sigmoid loss retrieves better (0.6602 versus 0.5547), and it holds a smaller edge at batch 64. One seed and linear encoders — read it as a demonstration of the claimed direction, not a measurement of its size.

People. Both papers came from large industrial teams rather than lone inventors. The ViT work was done at Google Research’s Brain team — Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Neil Houlsby, and colleagues, several of whom had built the big-transfer (BiT) CNN models that ViT was benchmarked against. CLIP was led at OpenAI by Alec Radford (also first author of GPT-1 and GPT-2) with Jong Wook Kim and others; Ilya Sutskever is among the senior authors. Radford’s three first-author systems (GPT, GPT-2, CLIP) share one design instinct: scale a simple objective on abundant data and let capability come from the data rather than the architecture.

What multimodal models enable

Once images and text share a space (or share a Transformer), several capabilities follow directly:

Open-vocabulary recognition. Classification is no longer bound to a fixed label set chosen at training time; classes are defined by text at inference. The same idea extends to detection and segmentation, where CLIP-style embeddings let models localize categories they were never explicitly trained on.
Cross-modal retrieval. Search images by text description, or find captions for an image, by nearest-neighbor lookup in the shared space — the mechanism of the retrieval@1 metric in our toy run, at the scale of billions of items.
Conditioning generation on language. Text-to-image systems use aligned text representations to steer generation: text encoders trained the CLIP way are part of models in the Stable Diffusion family, and CLIP embeddings were central to OpenAI’s DALL-E 2 pipeline. We meet the generative half of this machinery in diffusion models.
Vision-language assistants. A further step connects a vision encoder directly to a language model so it can converse about images: Flamingo (Alayrac et al., 2022) interleaves frozen vision and language backbones with trained cross-attention; open efforts such as LLaVA (2023) attach a CLIP-style vision encoder to a chat LLM with a small projection. This is the pattern behind commercial multimodal assistants, though public technical detail on those varies widely.

The through-line of this whole module is visible here. Tokenize everything (words as subwords, images as patches) and a single architecture with a scalable objective handles it. What remains genuinely different across modalities is mostly at the edges: how you cut the input into tokens, and what data teaches the model which structures matter.

Pitfall. The contrastive loss is defined over a batch, and its difficulty scales with batch size: distinguishing the right caption among 64 is far easier than among 32,768, the batch size the CLIP paper reports training with. Reproducing CLIP-style results on small hardware routinely yields great training loss and poor downstream embeddings, because tiny batches provide too few and too easy negatives, and a small toy run (like ours, with its 0.999 similarities) can look deceptively solved. When comparing contrastive training runs, hold batch size fixed or use methods designed for small batches (e.g., SigLIP’s sigmoid loss, Zhai et al., 2023), and evaluate on retrieval against a large candidate pool, not the training batch.

Fusion patterns: a taxonomy

“Multimodal model” covers architectures with very different couplings between vision and language, and being able to name the pattern tells you most of what a system can and cannot do. Four recur:

Late fusion (dual encoders). CLIP’s pattern: separate encoders per modality, interaction only through a similarity score in the shared space. Because each side can be embedded independently, you can precompute billions of image embeddings and search them with one text embedding — this is what makes web-scale retrieval and zero-shot classification cheap. The price: a single dot product is a thin channel, too thin for tasks needing detailed correspondence (“what number is on the jersey of the player on the left?”).
Cross-attention bridges. Flamingo’s pattern: keep a pretrained vision encoder and a pretrained language model frozen, and inject visual information into the language model through inserted cross-attention layers that let text tokens query image features. Richer than a dot product, cheaper than retraining either backbone, and naturally handles interleaved image-text sequences; the trained bridge is the only new component.
Projector into the token stream. LLaVA’s pattern (Liu et al., 2023): encode the image with a CLIP-style encoder, map the patch features through a small learned projection (a linear layer or MLP) into the language model’s embedding space, and hand them to the LLM as if they were a prefix of ordinary token embeddings. The LLM’s self-attention does all the fusion. This is the lightest-weight recipe (the projector can be trained with modest compute), and it is the dominant open-source pattern for chat-with-images assistants; BLIP-2’s Q-Former is a heavier cousin that compresses the image into a small set of learned query tokens first.
Early fusion (one token stream, one model). The most radical pattern: convert images into discrete tokens (via a learned quantizer) or raw patch embeddings, interleave them with text tokens in a single sequence, and train one Transformer over everything; models along the lines of Meta’s Chameleon report training this way from scratch. Fusion happens at every layer, generation of both modalities becomes possible in principle, and the clean separation between “vision encoder” and “language model” disappears — along with the ability to reuse pretrained unimodal parts.

The engineering trade-offs run along two axes: how much interaction the task needs (retrieval needs little, fine-grained grounding needs much), and how much pretrained machinery you can keep frozen (frozen backbones mean cheap training but limit how deeply the modalities can co-adapt). Reading a new multimodal system’s diagram, the first questions to ask are: where do the modalities first touch, and what was frozen when.

Going deeper

Vision pretraining without labels or captions. CLIP needs paired text; ViT-the-architecture does not. Masked autoencoders (He et al., 2022, “Masked Autoencoders Are Scalable Vision Learners”) drop a large fraction of patches (the paper uses 75%), encode only the visible ones, and train a lightweight decoder to reconstruct the missing pixels; the paper reports that this simple objective pretrains large ViTs to strong ImageNet fine-tuning accuracy, and the asymmetric design (encoder never sees masked tokens) makes pretraining fast. It is BERT’s masking idea translated to pixels, with the mask ratio cranked up because images are far more redundant than text. Self-distillation methods (the DINO line) learn strong features with no reconstruction at all, and DINO’s attention maps famously segment objects without any segmentation supervision — worth seeing once, with the register-token caveat from earlier in mind.

The CNN side did not stand still. ConvNeXt (Liu et al., 2022, “A ConvNet for the 2020s”) modernized a standard ResNet step by step (patchify-style stem, larger kernels, ViT-era training recipe) and reports accuracy competitive with comparably sized Swin Transformers across ImageNet and downstream tasks. Read alongside DeiT, it complicates any clean “attention beat convolution” narrative: much of ViT’s edge, at accessible scales, is attributable to training recipes that transfer back to CNNs. The honest current summary is that at moderate scale the two families are close, and the choice is driven by tooling, transfer sources, and the task’s structure.

What contrastive spaces miss. A CLIP-style space is shaped entirely by what captions mention. Attributes people rarely write down (exact counts, spatial relations, negation — “no people in this photo”) are weakly encoded, which is why CLIP-based systems fail benchmarks built on compositional distinctions. Follow-up datasets and losses target this, but the deeper lesson generalizes: a representation learns the distinctions its objective pays for, and “aligned with text” is not the same as “understands the scene.”

Open data, and what changed because of it. CLIP’s 400M pairs were never released, but open efforts (the LAION datasets among the best known) crawled and filtered public alt-text pairs at similar and larger scales, enabling open reproductions (OpenCLIP) and the text encoders inside open image generators. The same openness exposed the costs: web-scale image-text data carries duplicates, stereotypes, and in documented cases genuinely harmful content, and curation-versus-scale is now as live a debate in vision-language as the data-quality debates of the pretraining section.

Common pitfalls

Symptom: ViT patch embeddings “work” (shapes fine) but the model trains poorly, and visualizing reconstructed patches shows scrambled images. Cause: a wrong reshape/permute order in patchify — reshaping (B, C, H, W) directly to (B, n_patches, -1) interleaves rows from different patches. Fix: verify patchify with a round-trip test (unpatchify(patchify(x)) == x) before training, as in the script above.
Symptom: your from-scratch ViT badly trails a small CNN on a small dataset. Cause: not a bug — missing inductive biases cost accuracy in the low-data regime, matching the ViT paper’s own ImageNet-only results. Fix: use heavy augmentation and longer training (the DeiT recipe), start from pretrained weights, or accept the CNN for small-data problems.
Symptom: contrastive loss falls but embeddings are useless downstream; all vectors look alike. Cause: collapse or shortcut learning — often a missing normalization, a temperature stuck too high, or negatives that are trivially easy. Fix: normalize embeddings, make the temperature learnable (watch its value during training), and monitor matched-versus-mismatched similarity separately, as the script does.
Symptom: CLIP-style zero-shot accuracy varies by several points for the same model. Cause: prompt wording — “dog” versus “a photo of a dog” versus prompt ensembles changes text embeddings; the CLIP paper itself reports gains from prompt engineering and ensembling. Fix: fix and report the exact prompts; use the paper’s prompt templates when comparing to published numbers.

Try it yourself

Compare two patch-order conditions: (a) apply one fixed permutation to every training and test image, and (b) draw an independent random permutation per image. A sufficiently expressive MLP can relearn the fixed coordinate remapping; the per-example shuffle destroys a stable location-to-feature relationship. Measure and explain the difference without claiming that a fixed shuffle alone removes information.
Extend the patchify script to P = 8 and P = 16 on a 32x32 image, printing sequence length and per-patch dimension for each. What is the trade-off as P grows, and what happens to attention cost as P shrinks?
In the CLIP script, reduce the shared-concept dimension from 8 to 1 and rerun. What happens to retrieval@1 and to the matched-pair similarity, and why?
Break the pairing deliberately: shuffle txts so pair i’s caption belongs to a different image, and retrain. Where does the loss plateau relative to ln(64), and what does that tell you about what the loss measures?

Before class you should be able to

Compute the number of patch tokens from image size and patch size
Explain why patch embedding is exactly one strided convolution
State which inductive biases a CNN has that a ViT lacks, and what that costs in the low-data regime
Advise a small-data project: CNN or pretrained ViT, and why not a from-scratch ViT
Read the CLIP loss code and explain what the diagonal of the similarity matrix represents
Explain how CLIP performs zero-shot classification from typed class names
State the caveat that keeps an attention overlay from being a causal explanation

Connects to

HW3 (Transformer from scratch) is due today — this session is the final one of Module 3.
This session is covered by Quiz 3 (Module 3: S13–S22), given at S23.
Projects with images and limited data: the inductive-bias discussion here should directly inform your architecture choice, and the patchify code connects forward to diffusion backbones in Module 4.

← S21 · Alignment: SFT, RLHF, DPO · All readings · S23 · Autoencoders and VAEs →

How to read this

An image is a sequence of patches

Inductive bias versus data scale

The bias experiment on your CPU

Reading a ViT’s attention maps

CLIP: two encoders, one space

Temperature, and a sigmoid alternative

What multimodal models enable

Fusion patterns: a taxonomy

Going deeper

Common pitfalls

Try it yourself

Further reading

Before class you should be able to

Connects to