Word2Vec
The idea that turned words into arrows — and made meaning something you can measure with a dot product.
Hero generated with gpt-image-2 · sketchnote preset
Meaning from company kept
Word2Vec (Mikolov et al., 2013) takes that linguistic intuition and makes it a prediction task. Train a deliberately shallow network to predict a word from its context (or its context from the word), then throw the classifier away and keep the weights. Each row of that weight matrix is a dense vector — typically 100–300 dimensions — and words used in similar contexts land near each other in that space.
The shift is from symbol to geometry. One-hot vectors are orthogonal: cat and kitten are exactly as unrelated as cat and bureaucracy. Embeddings give you similarity, direction, and distance — the substrate every modern NLP model still stands on.
Context Window
A sliding window of size c defines "nearby". In the cat sat on the mat with c=2, the centre word sat pairs with {the, cat, on, the}. Small windows capture syntax; large windows capture topic.
Two Architectures
CBOW averages context vectors to predict the centre word — fast, better on frequent words. Skip-gram uses the centre word to predict each context word — slower, far better on rare words.
Negative Sampling
A full softmax over a 1M-word vocabulary is ruinous. Instead: score the true (word, context) pair as 1, and k = 5–20 random pairs as 0. Binary logistic regression replaces a giant normalisation.
The Embedding Matrix
Shape V×d (vocabulary × dimensions). It is not a means to an end — it is the product. Row i is the vector for word i. Cosine similarity between rows is your semantic ruler.
From corpus to coordinates
- Build the vocabulary from a large corpus; drop words below a minimum count. Optionally subsample very frequent words (the, of, and) — they carry little signal and dominate updates.
- Initialise two matrices at random: an input embedding (the vectors you'll keep) and an output embedding (context vectors, usually discarded).
- Slide the window across the text, emitting (centre, context) training pairs. This is self-supervised — the corpus is its own label source, no annotation needed.
- For each pair, draw k negative words from a unigram distribution raised to the 3/4 power — a tuned compromise that samples rare words more often than raw frequency would.
- Maximise
log σ(v·upos)plusΣ log σ(−v·uneg)by SGD: pull real pairs together, push sampled pairs apart. Only the handful of involved rows update — hence the speed. - After a few epochs, keep the input matrix. Normalise rows, and nearest-neighbour queries by cosine similarity now return semantically related words.
Choose CBOW when…
- Corpus is large and mostly common vocabulary
- Training speed matters most
- You want smoothed, averaged representations
Choose Skip-gram when…
- Corpus is smaller or domain-specific
- Rare and technical terms matter
- You want the higher-quality vectors (the usual default)
Semantic retrieval & query expansion. Match "affordable flat" to "cheap apartment" without a synonym list — the direct ancestor of today's vector databases.
Item2Vec. Treat a user's purchase or listen history as a "sentence" and products as "words". Airbnb, Spotify and Alibaba all shipped embeddings built this way.
Protein & gene embeddings. Amino-acid sequences as sentences: the same objective learns structural similarity from unlabelled sequence databases.
Log & clickstream analysis. Embed event types by co-occurrence to cluster failure modes and surface anomalous sequences.
Foundation for what followed. GloVe, fastText, then contextual embeddings (ELMo, BERT). Attention replaced the mechanism, not the premise.
Four honest limitations
- One vector per word, forever. bank (river) and bank (money) collapse into a single averaged point. This static-ness is precisely what contextual models later fixed.
- Similar ≠ same. Antonyms share contexts, so hot and cold sit close together. Cosine similarity measures relatedness, not agreement.
- Analogy arithmetic is oversold. It works on curated sets, and often only because the query word itself is excluded from the answer candidates. Treat it as a nice property, not a proof of reasoning.
- Bias is inherited, not invented. The corpus's stereotypes become geometric structure. If you deploy embeddings in a decision path, you deploy the corpus's prejudices with them.
Answer before you move on
Say them out loud or write two lines each. Retrieval beats re-reading.
- Why does Word2Vec discard the thing it was trained to do? Explain what the prediction task is really for. Hint: the objective is a scaffold; the weights are the deliverable.
- Negative sampling replaced the full softmax. What exactly was expensive about the softmax, and what does the approximation change about the loss being optimised? Hint: think about the denominator over V, and the shift from multiclass to many binary decisions.
- You must embed a small corpus of medical notes full of rare clinical terms. CBOW or Skip-gram — and what breaks if you pick the other one? Hint: which architecture gets more gradient signal per rare word?