Phase 6 · Deep Learning · #24

Long Short-Term Memory

The RNN that finally remembers what matters

Mental model: a conveyor belt with three valves. Memory rides straight along the belt; gates decide what to drop, what to add, and what to read out — so signal survives across long sequences instead of fading.

🎯 Core Concept

Plain RNNs multiply gradients through time, so error signal either vanishes or explodes over long sequences. The LSTM adds a protected cell state that information flows through with mostly additive updates — an uninterrupted gradient highway. Three learned gates regulate that highway, letting the network hold context for hundreds of steps.

Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ₜ

🔑 Key Components

Forget Gate

fₜ = σ(·) — decides what fraction of old memory to erase.

Input Gate

iₜ + tanh candidate — chooses which new info to write in.

Cell State

Cₜ — the long-term memory conveyor belt, additive path.

Output Gate

oₜ → hₜ = oₜ ⊙ tanh(Cₜ) — what to expose now.

⚙️ How It Works

Forget gate reads hₜ₋₁ and xₜ, outputs 0–1 mask over old memory.
Input gate + tanh candidate propose new content to store.
Cell state updates: keep some old, add some new (mostly addition).
Output gate filters the cell through tanh to produce the hidden state hₜ.

🌎 Real-World Applications

Machine translation Speech recognition Time-series forecasting Text generation Anomaly detection Handwriting recognition

🧪 Checkpoint Questions

1. Why does the additive cell-state update fix the vanishing-gradient problem that plagues plain RNNs? Hint: think about what happens to gradients under repeated multiplication vs. repeated addition.

2. If the forget gate outputs values near 1 for every step, what behavior would you expect from the cell state — and when is that useful? Hint: near-1 means memory is preserved almost untouched; consider very long dependencies.

3. When would you reach for a GRU or a Transformer instead of an LSTM? Hint: weigh parameter count / speed (GRU) and parallelism over long context (attention).