Phase 8 · Algorithm 33 of 30+

Attention Mechanisms

Let every position look back at every other position — and weight what actually matters.

Scaled dot-product attention: QKV projections, score matrix, scaling, softmax, weighted sum; plus multi-head, self vs cross attention, and the O(n-squared) cost curve.

Scaled dot‑product attention, end to end — plus the three things people get wrong about it.

The core idea

Before attention, a sequence model had to squeeze everything it had read into one fixed-size hidden state and carry it forward. Information decayed. Long-range links were lost.

Attention throws that bottleneck out. Instead of remembering, each position looks up what it needs, directly, from every other position — in a single parallel operation. Relevance stops being a function of distance and becomes a function of content.

Query, Key, Value

Querywhat this position is looking for
Keywhat each position advertises about itself
Valuewhat you actually take away when you match

All three are just learned linear projections of the same input: Q = XWq, K = XWk, V = XWv. The model learns how to ask, how to advertise, and what to hand over as three separate skills.

The whole mechanism in one line

Attention(Q,K,V) = softmax(QKT / √dk)V score → scale → normalise → weighted sum
  1. QKT — dot every query against every key. Similar directions score high. Result: an n×n compatibility matrix.
  2. ÷ √dk — without this, dot products grow with dimension, softmax saturates, and gradients vanish. One divide fixes a real training failure.
  3. softmax — turn raw scores into a probability distribution over positions: non-negative, sums to one. Now they are usable weights.
  4. × V — take the weighted average of the values. High-attention positions dominate the output; the rest fade.

Three things worth internalising

Multi-head is not redundancy

Split dmodel into h smaller subspaces, run attention independently in each, concatenate, project with Wo. Different heads reliably specialise — syntactic dependency, coreference, positional locality. One head can only express one relation per position; eight can express eight.

Self vs cross is only about where Q comes from

Self-attention: Q, K, V all from the same sequence — the sequence interrogating itself. Cross-attention: Q from one stream, K and V from another — a decoder querying an encoder. Translation, captioning, and retrieval-augmented models all live in that second case.

The cost is quadratic, and that shapes the field

Every token scores every token: n² computations, n² memory for the score matrix. Doubling context length quadruples both. Essentially all of modern long-context work — FlashAttention, sparse and sliding-window patterns, linear-attention families — is an attack on that one exponent.

Today's practice

Implement it from scratch — 30 minutes

  1. Write scaled_dot_product_attention(Q, K, V, mask=None) in NumPy. No frameworks. Shapes: (n, dk) in, (n, dv) out.
  2. Assert every output row of the softmax sums to 1.0. If it doesn't, you softmaxed the wrong axis — the most common bug in this code.
  3. Delete the √dk scaling and set dk=512. Print the max softmax weight. Watch it collapse toward 1.0 — that is saturation, felt rather than read.
  4. Add a causal mask (upper triangle to −∞) and confirm position i attends only to j ≤ i.
  5. Feed a real sentence's embeddings and print the attention matrix as a heatmap. Find one row where the peak is linguistically sensible.
Answer before you move on

Why does attention need three projections rather than two? Specifically: what breaks if you set V = K and reuse the keys as the values?