Autoencoders
Neural networks that learn to compress, then rebuild — distilling data down to its essence.
🎯 Core Concept
An autoencoder is trained to copy its input to its output — but through a narrow bottleneck that forces it to discard noise and keep only what matters. Because the targets ARE the inputs, no labels are needed: it is self-supervised. The magic lives in the middle: a compressed latent code that captures the data's underlying structure.
🧠 Mental Model
“A lossy zip file that learns its own compression scheme. Squeeze the data through a keyhole (the bottleneck) so tight that only the essential shape survives — then prove you kept the essence by rebuilding the original from it.”
🔑 Key Components
Encoder
Maps input x to a compressed code: f(x) = z. Layers shrink toward the bottleneck.
Bottleneck (z)
The low-dim latent space. Its size is the information budget — the tighter, the harder the distillation.
Decoder
Mirror of the encoder: g(z) = x̂. Expands the code back to full dimensionality.
Reconstruction Loss
Measures ||x − x̂||². Backprop trains encoder + decoder end-to-end together.
⚙️ How It Works
x → [Encoder] → z → [Decoder] → x̂ | minimize ||x − x̂||²
Feed input x through the encoder, shrinking it layer by layer into the latent code z.
The bottleneck z is far smaller than x — the network can't just memorize, it must compress.
The decoder expands z back into a reconstruction x̂ of the original size.
Compute reconstruction error and backpropagate through BOTH halves at once.
Over training, z becomes a meaningful, compact representation of the data manifold.
🌎 Real-World Applications
Dimensionality reduction
Anomaly detection
Image denoising
Data compression
Feature learning
Pretraining
Generative models (VAEs)
🧪 Checkpoint Questions
1. Why does the bottleneck HAVE to be smaller than the input? What would go wrong if the latent code were the same size as x?
Hint: think about what stops the network from learning a trivial identity (copy) function.
2. An autoencoder trained only on normal machine-sensor readings suddenly produces high reconstruction error on a new sample. Why does that make it a good anomaly detector?
Hint: it can only rebuild well what resembles its training distribution.
3. How does a linear autoencoder with squared-error loss relate to PCA (algorithm #14)? What does adding nonlinear activations buy you?
Hint: a linear AE recovers the same subspace as PCA; nonlinearity lets it bend to a curved manifold.