Causal Mask vs Padding Mask: The Two Attention Gates, and When Each Fires
One gate per role. No future leak. No filler attention.
Image backend outage note: the image generator (qwen/qwen-image-3 via OpenRouter) was funded but rendered a degraded plate on both rolls this run (garbled bullets, a duplicated column heading, missing bands, and the stable 768x1531 dimension deviation — the funded-primary degradation signature recorded Oct 10–14). Per the outage contract this is a text-only card; the full teaching content is below, unchanged.
1Causal mask: the no-future gate
Position-structured — the decoder's left-to-right law
In an autoregressive model, position i may attend only to positions 0..i.
Implemented as a lower-triangular bias matrix — same shape for every sequence.
During training it is the trick that makes parallel teacher-forced learning equivalent to sequential generation: every position is computed at once, yet each still sees only its past.
At inference the active step has only past tokens anyway, so the mask is redundant on the single active step — but the architecture still applies it.
It is what makes a decoder “causal” — without it, the model could copy future tokens during training and be useless at inference.
2Padding mask: the no-filler gate
Length-structured — wherever variable lengths are batched
Batching variable-length sequences pads the short ones to the batch maximum.
Padding tokens carry no meaning and must not enter any attention distribution.
Its shape changes with each sequence's real length — it is sequence-specific, not position-structured.
Fires in the encoder AND the decoder, in training AND inference — orthogonal to the encoder-vs-decoder question.
Keeps real tokens from attending to filler, and from being attended to by it.
3The one shared mechanism
What both gates actually are
Both masks are a bias addition to the attention score matrix — nothing more.
scores: S = QK^T / sqrt(d)
mask: S = S + bias, masked cells set to -inf
softmax: exp(-inf) = 0 → weight is exactly 0
The −inf is applied before the softmax, so the masked weight is not small — it is zero: attention that cannot exist.
The two gates compose: a GPT training batch needs both in one forward pass — causal so no future leaks, padding so no token looks at another sequence's filler.
4The trap the slides miss
The falsification — the correction to carry
“Zero embeddings would handle padding” is wrong. A zeroed embedding is not a mask at all.
A score of zero still gets non-zero probability mass after the softmax — it pollutes the distribution with attention that should not exist.
Only −inf (or a very large negative) makes the softmax actually output zero.
Test any explanation: if it says “set the score to zero” instead of “set it to −inf before softmax”, it is describing a model that still attends to the future and the filler.
The line to remember
A mask is a bias in. -inf before softmax. Zero weight out.