Pre-Norm vs Post-Norm: How a Clean Residual Highway Lets GPT Train 96 Layers

• Four teaching sections on a notebook page:
• WHY NORMS EXIST
• After every attention/FFN block a token vector drifts: magnitude grows, scale shifts with depth
• Norm re-centers each token: subtract mean, divide by std, so the next sublayer sees stable magnitudes no matter how deep the stack
• It is a per-token internal stabilizer, NOT the input feature scaling of classical gradient descent
• LN vs BN: WHICH DIRECTION YOU MEASURE
• BatchNorm: mean/std across the BATCH — same feature, many samples; needs a real batch, breaks at inference with batch size 1
• Padded variable-length sequences corrupt the batch stats; a causal model cannot peek at future positions
• LayerNorm: mean/std across ONE token's own features; no batch dependence, works token by token: exactly what autoregressive inference is
• POST vs PRE: WHERE THE NORM SITS (two side-by-side boxes, keep them opposite)
• POST-NORM box (2017): x + Norm(Sublayer(x)). The gradient must PASS THROUGH the norm to reach the residual. Early training squashes, needs warmup, struggled past about 12 layers
• PRE-NORM box (modern default): x + Sublayer(Norm(x)). The gradient BYPASSES the norm entirely: a clean residual highway straight down to layer 1, so 96+ layers train stably at a constant learning rate
• THE TRADE-OFF AND THE PROOF
• Pre-norm's output skips the last norm, so the final representation is slightly weaker
• That is why GPT and Llama add one final LayerNorm after the last block
• Falsification: post-norm needed warmup and could not pass about 12 layers; pre-norm scales to 96+ layers at constant learning rate