One Word With One Vector Cannot Mean Two Things
Static embeddings give every word exactly one point in space. That one design choice buys you arithmetic and costs you ambiguity.
Core Concept
word2vec and GloVe assign each word ONE vector, learned from which words co-occur across a corpus. That single choice is the whole story.
It buys you real geometry — king - man + woman lands near queen. And it costs you polysemy: bank gets exactly one point in space, so training pulls it toward both the riverbank sense and the financial sense, and settles on a compromise that is faithful to neither.
A contextual model — ELMo, then BERT and every transformer since — computes the vector at inference time from the surrounding tokens, via self-attention. So bank in "river bank" and bank in "bank loan" are different vectors from the same weights.
Key Components
One vector per word
Static embeddings are a lookup table. The vector is fixed the moment training ends and never varies by sentence.
Co-occurrence geometry
Vectors learned from context windows across a corpus. The reward: meaningful directions you can do arithmetic on.
Polysemy collapse
Two senses, one point. The optimiser averages them into a location that serves neither sense well.
Inference-time vectors
Self-attention conditions each token on its neighbours, so the same word yields different vectors per sentence.
How It Works
- Static: scan the corpus, learn one vector per vocabulary item from co-occurrence statistics, freeze the table.
- Lookup: at use time you index a dictionary. Microseconds, deterministic, cacheable forever.
- Contextual: feed the whole sentence through the model; self-attention mixes each token with its neighbours.
- Forward pass required: the vector only exists once the sentence exists — so you cannot precompute one per vocabulary word.
The Cost Is Symmetrical
| Dimension | Static | Contextual |
|---|---|---|
| Retrieval | Dictionary lookup | Forward pass per sentence |
| Latency | Microseconds | Model inference cost |
| Precompute | Whole vocabulary, once | Impossible — depends on context |
| Ambiguous words | One compromised point | Distinct vectors per sense |
The Falsification — Read This Twice
If your unit of meaning is a single word with no sentence around it — a tag, a product SKU, a query term — there IS no context to condition on. You are paying a transformer to reproduce a lookup.
And cosine similarity between contextual vectors is not the same measurement. Two occurrences of the same word in different sentences are legitimately far apart. Code that clusters them expecting one point per word gets a cloud instead — and the bug looks like bad data, not a category error.
Where This Decides the Design
- Tag & SKU embeddings — isolated tokens with no surrounding sentence. Static is not a compromise here; it is the correct fit.
- Semantic search over documents — sentences carry context, and disambiguating "bank" or "apple" changes which results are right.
- Word-sense disambiguation & NER — the entire task is that one surface form means different things. Static cannot represent the question.
- Latency-bound autocomplete — a cached lookup table beats a per-keystroke forward pass on cost and tail latency.
- Clustering pipelines — the place teams silently break things by swapping a static table for a contextual model and keeping the old cosine assumptions.
Checkpoint — answer before you move on
- Why does a single vector for "bank" end up faithful to neither the river sense nor the money sense? Think about what the optimiser is averaging over during training.
- Name the concrete operational cost of contextual embeddings that static ones do not have — and say why it rules out one common optimisation. What can you build ahead of time for a whole vocabulary, and what can't you?
- You swap a static table for BERT in a clustering job and the clusters fall apart. What assumption broke? How many points per word did the old code expect?