Learning From Data With No Labels
Subtitle: The labels come from the data.
Top band - The move: Invent a task out of the data's own shape. The labels are free.
Left column - Masked prediction:
Hide a word, predict it Hide a patch, rebuild it Label = the hidden part Learns from one view Family: BERT, Word2Vec
Right column - Contrastive:
Two crops of one image Pull the pair together Push other images apart Label = same or not same Family: SimCLR, CLIP
Simple difference:
Masked: fill in the blank Contrastive: match the pair
Use it when - Masked prediction: text, speech, dense pixel detail, long context. Use it when - Contrastive: whole-image meaning, search and retrieval, few classes.
Band - The real deliverable: Throw the pretext task away. Keep the weights. This scales with raw data, not with labelling budget.
- Honest test - three steps:
- Freeze the encoder
- Train a linear probe
- Small labelled set decides
Sticky note - Common beginner mistake: A low pretext loss proves nothing. A shortcut can win the task and teach no meaning - edges alone can reassemble patches. Judge it downstream.