Algorithm 35 of 40 · Phase 8

GPT Architecture

The decoder-only Transformer: one objective, predict the next token, scaled until it becomes general capability.
Generative Pre-trained Transformer Autoregressive Causal masking Style: technical-poster
GPT architecture infographic: stacked decoder blocks, causal attention mask, next-token prediction objective and scaling law
Decoder-only stack · causal mask · next-token objective · scaling behaviour
Core Concept

GPT throws away half of the original Transformer. No encoder, no cross-attention — just a tall stack of identical decoder blocks reading left to right. Every position may attend to itself and everything before it, never to the future. That single constraint is what turns a sequence model into a generator.

Training has exactly one job: given tokens x1…xt-1, predict xt. No labels, no task heads — the internet itself is the supervision. Because the causal mask lets every position be predicted in parallel from one forward pass, a sequence of length T yields T training signals at once. That efficiency, plus scale, is the whole story.

maximize  ∑t log P(xt | x<t; θ) Next-token likelihood. Deceptively simple — yet to lower this loss the model must learn syntax, facts, arithmetic, style and rough reasoning, because all of them reduce prediction error.
Attention(Q,K,V) = softmax( QKT/√dk + M )V M is the causal mask: 0 on and below the diagonal, −∞ above it. After softmax those become exactly 0 — the future is unreachable, not merely discouraged.
Key Components
01

Token + Position Embeddings

Tokens map to dmodel vectors and are summed with positional information. Attention is permutation-invariant on its own — without this, "dog bites man" and "man bites dog" are identical inputs.

02

Masked Multi-Head Self-Attention

Each head learns a different relation (nearby syntax, coreference, long-range topic) in its own subspace. The lower-triangular mask enforces causality so the model can never peek at the answer it is being asked to predict.

03

Feed-Forward + Residual + LayerNorm

A per-position MLP (typically 4× wider, GELU) does the non-linear "thinking" between attention rounds. Pre-norm residual paths keep gradients alive through dozens or hundreds of layers.

04

LM Head & Sampling

A final linear layer — often weight-tied to the embedding matrix — projects to vocabulary logits, then softmax. Temperature / top-p / top-k decide how boldly you sample from that distribution.

How It Works
  1. Tokenize and embedText becomes subword IDs (BPE), each ID becomes a vector, position is added in. You now have a T × d matrix.
  2. Attend, causallyProject to Q, K, V per head. Score, scale by √dk, add the −∞ mask, softmax, then take a weighted sum of values. Each token gathers context strictly from its past.
  3. Transform per positionFeed-forward network refines each position independently. Residual connections mean every block edits a shared representation rather than rebuilding it.
  4. Repeat N timesEarly layers look local and syntactic; middle layers carry semantics and entities; late layers sharpen toward the actual next-token decision.
  5. Predict, in parallel, everywhereOne forward pass produces a distribution at every position. Cross-entropy against the true next token; backprop once. This is why pre-training is compute-bound, not label-bound.
  6. Generate autoregressivelyAt inference: sample a token, append it, feed it back. KV-caching stores past keys/values so step t costs O(t) instead of re-running the whole prefix.
Real-World Applications
  • Conversational assistants — ChatGPT, Claude, Gemini. Base next-token model, then instruction tuning and RLHF/DPO to align behaviour.
  • Code generation — Copilot, Codex, Cursor. Code is unusually well-suited: strongly structured, and causal prediction matches how you type.
  • Agents & tool use — the same loop emits function calls; the environment's response is appended as more tokens. Reasoning becomes generation.
  • Long-form drafting & translation — summarization, rewriting, cross-lingual transfer, all as conditional continuation of a prompt.
  • Beyond text — decoder-only stacks now model audio, protein sequences, robot action tokens and video frames. The architecture is modality-agnostic.
Sensei's note: The interesting thing about GPT is not its cleverness — it is its austerity. One block, repeated. One loss, next-token. Almost every capability you associate with these models is an emergent side-effect of compressing the world well enough to predict it. Hold that thought when you are tempted to add complexity to your own designs.
Checkpoint — answer before you move on
  1. Why does GPT drop the encoder and cross-attention that the original Transformer had — what does the decoder-only design buy you? Hint: think about what the encoder was for (a separate source sequence), and what "prompt + continuation as one stream" makes possible.
  2. What exactly does the causal mask prevent, and why would training silently break without it rather than just get worse? Hint: trace what the loss would be at position t if the model could attend to position t.
  3. Training scores every position in a single forward pass, yet generation must run one token at a time. Explain that asymmetry — and what KV-caching does about it. Hint: at training time the true tokens already exist; at inference they don't.