Grouped Query Attention: Why Every Query Head No Longer Needs Its Own KV Cache
Share the K/V. Shrink the cache. The decoder’s memory budget, made explicit.
Image backend outage note: the image generator (qwen/qwen-image-3 via OpenRouter) was funded (exit 0, watermark OK, no 402) but rendered a degraded plate on the roll this run (title word Owne, panel headings Withel cerchg / The growinging, corrupted bullet lines, prompt-scaffolding text sstir.Ing on the sticky note — the same funded-primary degradation signature recorded on Oct 10–13). Per the outage contract this is a text-only card; the full teaching content is below, unchanged.
1Where GQA lives
Decoder attention blocks only — the side that generates token by token
GQA changes the K and V projections in each decoder attention block — nothing else.
Query heads keep their full count (say 8); only the key/value heads shrink.
It exists where the KV cache lives: inference, generating one token at a time.
A T5-style encoder scores the whole input in one batch pass — no incremental generation, no KV cache at all to shrink.
So “Llama 3, Mistral, Gemma, Falcon use GQA” is true for the decoder stack only — never let a card imply the encoder has a KV cache.
2The grouping
One axis: how many KV heads the query heads share
MHA: every Q head owns its own K and V — 8 query heads, 8 KV pairs, the heaviest cache.
GQA: a group of Q heads shares one K/V pair — 8 query heads, 4 KV pairs, the balanced point.
MQA: all Q heads share a single K/V pair — 8 query heads, 1 KV pair, lowest memory.
KV heads: 8 → 4 → 1
quality: MHA > GQA > MQA
Query heads stay full in all three — the model’s “eyes” are never reduced, only what each eye’s cache must store.
GQA is the sweet spot the modern defaults picked: most of MHA quality, a fraction of the cache.
3What it buys
The arithmetic the memory argument rests on
KV-cache bytes per token scale with the number of KV heads, not the number of query heads.
Same 8 query heads, one KV head instead of eight: the cache is 8x smaller.
At long context the KV cache can exceed the model weights in memory — the context, not the model, becomes the memory bill.
That gap is the line between serving 128K context on one GPU and not serving it at all.
4Honest test
The falsification — test any GQA explanation against these
Ask a T5-style encoder: does it hold a KV cache while scoring the full input? No — nothing is generated incrementally, so there is nothing to cache.
Any claim that “GQA halves the encoder’s memory” is wrong on the spot — GQA has no encoder to act on.
Check a real model card: KV heads are listed explicitly and are always fewer than query heads (e.g. 8 vs 4).
GQA exists only where tokens come one by one; if your workload scores whole inputs in one pass, the optimization is empty space.