AdamW: How Momentum and Per-Parameter Learning Rates Steer LLM Training

Panel 1 (full-width horizontal flow diagram, five boxes with arrows, short labels): BATCH -> MODEL FORWARD (tokens to a loss number) -> LOSS (one number) -> BACKPROP (chain rule: one gradient per weight) -> ADAMW (optimizer: gradients become weight updates) -> NEW WEIGHTS, with a dashed loop arrow from NEW WEIGHTS back to BATCH labeled "repeat, next batch, thousands of times".

• Panel 2 (heading "The gap the recap card elides", short bullets):
• The model only maps tokens to a loss; it learns nothing alone
• Backprop is information, not motion: per-weight direction and size
• The optimizer is the missing verb: gradients become weight updates
• Without it, backprop alone changes nothing

• Panel 3 (heading "Adam: two running averages", short labeled lines):
• m = EMA of the gradients (momentum)
• v = EMA of squared gradients (RMS)
• m smooths noise; v measures how noisy each weight usually is
• w = w - lr * m / (sqrt(v) + eps)
• Divide by sqrt(v): each weight gets its own effective step size

• Panel 4 (heading "Why the W, and the honest test", short labeled lines):
• Plain Adam: decay baked into the gradient, distorted by the scaling
• AdamW (2019): decay is a separate step on the weights: w = w*(1 - wd*lr)
• Decoupled = clean, uniform decay; stabilizes training at LLM scale
• Falsify: skip the update, same batch: loss does not move