Core Concept
Supervised learning needs someone to say "this is a cat." Contrastive learning asks a cheaper
question: which two of these are the same thing? Take one image, distort it twice — that pair
must land close together in embedding space. Every other image in the batch must be pushed
far apart. The labels come free, generated by the augmentations themselves. Do this on millions
of unlabelled images and the encoder learns features so good that a single linear layer on top rivals
fully supervised training.
Key Components
01Augmentation Pipeline
Random crop, colour jitter, grayscale, blur, flip. This is the supervision signal — weak augmentation makes the task trivial, so the model learns nothing but colour histograms.
02Twin Encoder + Head
One shared backbone f (e.g. ResNet/ViT) encodes both views, then a small MLP
projection head g maps to the space where loss is computed. The head is thrown away
afterwards; you keep f.
03Positives & Negatives
Positive = two views of the same source. Negatives = every other sample in the batch or queue. More negatives ⇒ a harder, more informative task.
04InfoNCE Loss + τ
A softmax over cosine similarities: "pick the true partner out of N candidates." Temperature
τ controls how hard the model punishes near-miss negatives.
NT-Xent / InfoNCE loss for a positive pair (i, j)
ℓ(i,j) = −log [ exp(sim(zᵢ,zj)/τ) / Σk≠i exp(sim(zᵢ,zk)/τ) ]
sim(u,v) = uᵀv / (‖u‖‖v‖) — cosine similarity on the unit hypersphere.
Numerator = the one true positive. Denominator = positive + all 2N−2 negatives.
Small τ (≈0.07–0.2) sharpens the distribution and focuses gradient on the hardest negatives.
How It Works
- Sample a batch of N unlabelled images.
- Apply two independent random augmentations to each → 2N views, N positive pairs.
- Encode all views with the shared backbone, project to L2-normalised embeddings
z. - Build the 2N × 2N cosine-similarity matrix; mask the diagonal (self-similarity).
- Minimise InfoNCE: pull each positive pair together, push all other entries in its row apart.
- Discard the projection head. Freeze the backbone and fit a linear probe on a small labelled set to measure representation quality.
Family Tree — Same Idea, Different Negatives
| Method | Where negatives come from | Trade-off |
|---|---|---|
| SimCLR | Other samples in a very large batch | Simple; needs batch 4096+ (TPU-scale memory) |
| MoCo | Momentum-updated queue of past embeddings | Huge negative set on modest GPUs; extra momentum encoder |
| BYOL / SimSiam | None — predictor + stop-gradient asymmetry | No negatives at all; relies on architectural tricks to avoid collapse |
| CLIP | Cross-modal: other captions in the batch | Image↔text alignment ⇒ zero-shot classification |
| SupCon | Labels define positives (all same-class samples) | Supervised variant; beats cross-entropy on robustness |
Real-World Applications
🖼️Pretraining without labels
Medical imaging and satellite data, where expert annotation is the bottleneck — pretrain on all of it, fine-tune on the labelled 1%.
🔎Semantic search & retrieval
Sentence embeddings (SimCSE, E5) and image search: contrastive training is what makes nearest-neighbour lookup mean anything.
🗣️Multimodal alignment
CLIP-style image↔text training powers zero-shot classification and the text conditioning inside diffusion models (Card 36).
🎧Speech & audio
wav2vec 2.0 contrasts masked latent frames against distractors to learn speech units before any transcript is seen.
🛡️Face / speaker verification
Same identity → close; different identity → far. The metric-learning ancestor of all of this.
The failure mode to know: representation collapse — the encoder maps everything to one
constant vector, which trivially minimises the "pull together" term. Negatives are what prevent it.
Remove them (BYOL/SimSiam) and you must replace them with a stop-gradient plus predictor asymmetry,
or training silently collapses to a useless constant.
Checkpoint — answer before you move on
Three questions
- Where does the training signal come from if there are no labels? Hint: what defines a positive pair, and what would happen if you removed augmentation entirely?
- What does the temperature τ in InfoNCE actually control, and what breaks at τ → very large or very small? Hint: think about the sharpness of the softmax over similarities and which negatives dominate the gradient.
- SimCLR needs enormous batches; MoCo doesn't. What did MoCo change, and why does BYOL need neither? Hint: name where each method's negatives live — and what stops BYOL from collapsing.
Phase 8 progress · 38 of 50 algorithms · style: sketchnote