Semantic Embeddings: How Context Proximity Shapes Vector Space

Meaning is geometry: co-occurrence trains the distance, and one word can hold one vector — or a fresh one for every sentence it meets.

Static: one vector per word

• An embedding layer is a learned lookup table — 50,000 × 512 rows for a 50,000-word vocab, one row per token, shared across every sentence.
• Training is co-occurrence: rows drift closer as words appear in similar contexts, so cat and kitten land nearby while cat and turbine sit far apart.
• The price: a word that means two things gets one compromise vector — the static bank is an average of river-bank and money-bank, neither fully.

Contextual: one vector per occurrence

• A transformer emits a new vector for every occurrence: the representation is a function of the whole surrounding sentence, not a table row.
• Bank in “the river flooded its bank” and bank in “I deposited at the bank” now separate — polysemy is solved, not averaged away.
• The distance geometry emerges from attention over context at read time; nothing is pre-baked per word.

The algebra — and who it belongs to

• king − man + woman ≈ queen: the arithmetic works because static vectors bake syntactic relations into their axes — a property of the model class, not of the words.
• On contextual vectors the axes dissolve (the vector is sentence-bound, not word-bound), so the clean analogy algebra is a static-model property.
• Split the trade: static wins algebra; contextual wins polysemy and retrieval. Royalty cluster (king/queen) vs machinery cluster (turbine/wrench): large semantic distance, and the gap is the lesson.

Falsify the framing

• One token, two sentences, two vectors: embed “bank” in both sentences from the same contextual model. If the two vectors are identical, the contextual claim is false.
• Distance check: cosine(river-bank, deposit-bank) must be visibly lower than cosine(river-bank, river-bed) — if not, context is not actually doing the work.
• Try the algebra on contextual vectors: king − man + woman on sentence-bound embeddings should fail to land near “queen”. If it still works cleanly, you are really holding static vectors and calling them contextual.
• Table or attention: if you can dump a per-word table that predicts the model’s “contextual” vectors without the sentence, the representation is not contextual.